README.md

---
license: apache-2.0
base_model: google/gemma-4-E2B
library_name: peft
language:
- en
tags:
- web-agents
- gui-agents
- mind2web
- lora
- pointer-network
- selective-prediction
- calibration
- decision-making
- fast-decision-model
- system-1
- model-routing
- gui-grounding
datasets:
- osunlp/Multimodal-Mind2Web
---

# Gemma 4 E2B + LoRA pointer head for web element selection (Multimodal-Mind2Web)

**A small, fast "which element do I click next?" model with a confidence score. Research release, not a ready-to-use web agent.**

Given a task, the previous actions, a full-page screenshot and a list of candidate DOM elements, the model points at one candidate (or at a learned "none of these" slot) and predicts the operation type (click / type / select). It does **not** generate the text to type.

- Backbone: `google/gemma-4-E2B` (frozen) + LoRA (rank 32, attention + MLP, layers 0-29, 180 modules) + a small pointer head trained from scratch.
- Trained on Multimodal-Mind2Web for about 8 hours on one A100-40GB.
- Speed: in the live form test below (short pages, one screenshot strip, about 33 candidates) a decision took about 0.2 s for the model plus 0.1 s for candidate extraction and screenshot on an A100. Inputs of Mind2Web size (several strips, up to 4600 tokens) are slower; we did not benchmark that separately.
- Honest summary of what it is good and bad at is in [Summary of results](#summary-of-results) and [Limitations](#limitations-and-what-this-model-is-not).

**Role: a fast decision module ("System 1").** The model is meant as the quick, cheap first decision in a larger agent: it picks the next element or abstains with a confidence score, and low-confidence steps are escalated to a stronger model ("System 2", for example a large VLM) or a human. It is a decision/routing component, not a standalone agent.

**Approach: a calibrated decision head, "Kev-like".** Instead of generating the next action as text, the model makes a *decision over a closed set*: a query vector (from the task, history and screenshot) is scored against one key per candidate element plus a learned "none of these" key, and a softmax gives a probability for every option. This pointer-style, "Kev-like" decision head has three consequences we care about: (1) one forward pass picks the element, no token-by-token decoding; (2) the top-1 probability is a ready-made confidence score, and "none" is an explicit option, so the model can abstain; (3) because decisions and confidences are explicit numbers, they can be evaluated and calibrated with standard tools (ECE, Brier score, risk-coverage curves, temperature scaling) and used to route uncertain steps to a stronger model. In this release the decision quality and the confidence ranking are measured (see results); the confidences themselves are still raw and need temperature scaling on held-out data before being read as probabilities. The "Kev-like" design follows [Kev](https://github.com/jaredpalmer/kev) by Jared Palmer (open-source family of "System One" decision models: a small LoRA adapter plus a pointer head on a Qwen base, [weights on Hugging Face](https://huggingface.co/collections/jaredpalmer/kev), Apache-2.0), which in turn is described as Jev-like; see [Related work](#related-work-and-credit).

> **Status:** single training run, single seed. Confidence scores are **raw (not temperature-scaled)** for this checkpoint. Numbers are not directly comparable with published Mind2Web numbers (see [Evaluation protocol](#evaluation-protocol)).

## Summary of results

Element accuracy on Multimodal-Mind2Web test splits (`n` = steps; 95% bootstrap CI). A step counts as correct if the top-1 candidate is a ground-truth element. A "none" prediction counts as an error.

| Split | n | Accuracy (CI95) | Accuracy if the answer is in the list | Filter ceiling | ECE (raw) | AUROC of confidence | Acc. on most confident 20% |
|---|---|---|---|---|---|---|---|
| test_task | 1339 | **0.465** [0.439, 0.493] | 0.556 | 0.837 | 0.189 | 0.710 | 0.761 |
| test_website | 1019 | **0.384** [0.355, 0.413] | 0.483 | 0.795 | 0.189 | 0.698 | 0.662 |
| test_domain | 4060 | **0.385** [0.370, 0.400] | 0.456 | 0.845 | 0.199 | 0.737 | 0.703 |

"Filter ceiling" is the fraction of steps in which our rule-based candidate filter keeps a correct element at all (see below). Nobody can score above it with this candidate list. Validation (579 steps carved from the training websites): accuracy 0.368 [0.326, 0.408].

Progression across our own runs (same candidate filter, same splits):

| Run | test_task | test_website | test_domain |
|---|---|---|---|
| First text-only run, LoRA r16 attention | 0.375 | 0.275 | 0.308 |
| + full-page screenshot, LoRA r16 attention, 24-token candidate cap | 0.416 | 0.317 | 0.351 |
| **This model**: screenshot, LoRA r32 attention+MLP, tree-format candidates, gold candidate always present in training | **0.465** | **0.384** | **0.385** |

Risk-coverage (accuracy if the model only answers its most confident x% of steps; raw confidences):

| Split | 100% | 80% | 60% | 40% | 20% |
|---|---|---|---|---|---|
| test_task | 0.465 | 0.515 | 0.583 | 0.660 | 0.761 |
| test_website | 0.384 | 0.426 | 0.471 | 0.574 | 0.662 |
| test_domain | 0.385 | 0.445 | 0.513 | 0.591 | 0.703 |

Other metrics (test_task / test_website / test_domain): operation-type accuracy 0.881 / 0.866 / 0.864; Brier 0.256 / 0.249 / 0.240; fraction of steps where the model answers "none" 0.120 / 0.119 / 0.092; "none" predicted when no correct element is in the list (abstention success) 0.367 / 0.373 / 0.237.

**Take-aways**

1. The screenshot helps, and the gain is driven by the pixels. In an ablation on the earlier vision run (not repeated for this checkpoint), replacing the screenshot with a gray image or another page removed the gain (paired CI of the gain over the text-only model: +3.2 to +5.4 points).
2. The confidence score carries information: accuracy rises from 0.38-0.47 on all steps to 0.66-0.76 on the most confident 20%. This is the main reason to use this model (selective prediction, routing hard cases to a stronger model), not its raw accuracy.
3. Raw confidences are over-confident (ECE about 0.19-0.20). On the earlier vision model temperature scaling fitted on validation reduced ECE to roughly 0.05-0.09; **we have not measured this for this checkpoint**. Fit a temperature on your own held-out data before using the numbers as probabilities.
4. Absolute accuracy is modest. About 15-20% of steps lose the correct element in our candidate filter, and even when the correct element is in the list the model finds it only 46-56% of the time.

## Intended use

- Research on small, fast GUI/web element grounding with calibrated abstention.
- A cheap first-pass element selector whose low-confidence steps are escalated to a larger model or a human (cascade/routing experiments).
- A baseline and a set of documented negative results for pointer-style heads on top of a small VLM.

## Model details

**Architecture.** The full Gemma 4 E2B model (35 layers, hidden size 1536) runs on the whole sequence; we read the hidden states of **layer 30** and attach:

- a **query** from the last token (LayerNorm -> linear, d_k = 256),
- one **key per candidate** from the mean of the candidate's token hidden states (LayerNorm -> linear),
- a learned **"none" key**, so the softmax is over `n_candidates + 1` slots,
- an **operation head** (linear, 3 classes: CLICK / TYPE / SELECT) on the query.

LoRA is applied to layers 0-29 (q/k/v/o and gate/up/down projections where they exist; the last 20 layers of Gemma 4 E2B share KV and have no k/v projections), rank 32, alpha 64, dropout 0.05. The vision tower and the language model weights are frozen. Hidden states of layers 30-34 are not used by the head (they are still computed in the forward pass; truncating the model at layer 30 would be an easy speed-up we did not implement).

**Input.** `[BOS] [screenshot strips] Task: ... Previous actions: ... Candidates: - <candidate 1> - <candidate 2> ... Task: ... Previous actions: ... Next element to interact with:`

- *Screenshot*: the full-page screenshot is cut top-to-bottom into 1280x1280 strips (a trailing piece under 200 px is merged), at most 5 strips (6400 px), 280 vision tokens per strip. About 19% of Mind2Web pages are taller than that and are truncated at the bottom. No target-dependent cropping is used (it would leak the answer).
- *Candidates*: text only, in "tree" format: `role "accessible-ish name" [state]`, for example `textbox "Where from?"`, `link "Flights"`, `button "Search"`. At most 64 tokens per candidate and 4600 tokens in total per step (candidate lines are shortened uniformly when a step is too long). **The model is not given element positions**, so it must match candidate text to what it sees in the screenshot.
- *Candidate list*: Multimodal-Mind2Web provides 331-743 candidates per step. We order them by TF-IDF similarity to the task and recent actions, then apply a rule filter (visible, interactive tags first, then text-bearing common tags) and keep at most K = 160.

**Loss.** Multi-positive softmax cross-entropy over candidates plus "none" (target is "none" if no correct candidate is in the list), plus 0.3 x cross-entropy for the operation type.

## Training

| | |
|---|---|
| Data | Multimodal-Mind2Web `train` (7775 steps): 7196 for training, 579 for validation (validation = websites whose `crc32(website) % 8 == 0`) |
| Candidate handling in training | rule filter K = 160, then the dropped correct candidate is **added back** to the list (`--always_pos`). Validation and test lists are never modified |
| Epochs | 4 run, **checkpoint of epoch index 2 (third epoch) kept** (best validation accuracy: 0.249 / 0.363 / **0.368** / 0.358; ECE rose 0.119 -> 0.164 -> 0.260 -> 0.362, i.e. over-fitting in the last epoch) |
| Optimiser | AdamW, LoRA lr 2e-4, head lr 1e-3, cosine schedule, 3% warm-up, gradient clipping 1.0, micro-batch 1 with accumulation 8, bf16 |
| Candidate order | random permutation in training; original TF-IDF order at test time |
| Hardware / time | 1x A100-SXM4-40GB, 29,604 s for 4 epochs (about 1.03 s per example), peak memory 38.6 GB |
| Skipped | 4 training steps skipped for out-of-memory (inputs of about 13.8k tokens); 11 steps without a screenshot |
| Seeds | one (seed 0) |
| Base model revision | `google/gemma-4-E2B` @ `d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f` |

Before this run we swept LoRA capacity, learning rate, candidate text format, and the query layer on a 1500-example, 2-epoch text-only setup (val 579 steps, single seed): rank-32 attention+MLP was the only clear winner (about +6.6 points on validation over rank-16 attention-only); rank 64, lr 1e-4 and 4e-4, and query layer 35 were not better; the tree format was as accurate as the long attribute format and shorter; a 24-token candidate cap did not measurably hurt in the short regime. These differences are from one seed and a small validation set, treat them as hints.

## Evaluation protocol

- Splits: the official Multimodal-Mind2Web `test_task`, `test_website`, `test_domain`.
- Metric: top-1 element accuracy over **our** candidate list (K = 160, rule filter). This is **not** the official candidate list (the Mind2Web paper uses a DeBERTa ranker's top-50), and steps with no correct element in the list count as errors. Our numbers must not be compared with published element accuracy without this caveat; we make no claim of parity or superiority with any published system.
- "Accuracy if the answer is in the list" is conditional on the correct element being among the candidates and is **higher by construction**; it should not be compared with unconditional numbers.
- ECE uses 15 equal-width bins on the top-1 probability. AUROC measures how well the top-1 probability separates correct from incorrect steps.
- Bootstrap confidence intervals are over steps. Several steps come from the same task or website, so the true uncertainty is somewhat larger than the CIs suggest.

## A stress test outside the training distribution: live form filling (negative result)

We ran the model in a real browser (Playwright) on 12 synthetic forms and, when reachable, 5 public practice pages (no submit buttons were pressed). One step per field; the field label and value come from the task sentence; the model only picks the element. We compare two candidate styles: **as in the dataset** (raw attributes, no `<label>` binding) and **with labels** (label text of the associated `<label>` added as the accessible name), and a trivial no-model baseline (TF-IDF match of the field label against candidate text; this baseline is told the target label directly).

| Setting (86 steps, 17 forms) | Step accuracy (history = ground truth) | Fields correctly filled when the model drives (free run) | Forms fully correct |
|---|---|---|---|
| Gemma pointer, dataset-style candidates | 0.36 | 0.29 | 2/17 |
| Gemma pointer, label-bound candidates | 0.55 | 0.49 | 3/17 |
| Lexical-match baseline (knows the label) | 0.78 | 0.70 | 6/17 |

So: **this model is not a usable form filler.** What we found:

- Candidate lines like `textbox "fld3"` (opaque id/name) cannot be matched with the visible label, because the model never sees element positions. Adding the label text gave +19 points. Placeholder-based forms (text is in the candidate) scored 0.77, forms whose label is an unbound `<div>` 0.21.
- By type: text 0.50-0.75, select 0.36-0.57, checkbox/radio close to 0.
- The model often picks `button "Submit"` for fields that are not done yet, with 96-99% confidence (a learned "what comes next" prior). Confidence ranking is still informative overall (label-bound free run: AUROC 0.88, 0.94 accuracy on the most confident 20%), but confidently wrong steps exist.
- The operation head dropped from 0.88 on Mind2Web to 0.56-0.63 here: the task phrasing (many fields in one sentence) and the pages differ from the training distribution.
- 86 steps is a small sample: differences of about 10 points are within noise.

## Limitations and what this model is not

- **Not an agent.** It selects an element and an operation type. It does not produce typed text, does not plan, and does not recover from errors.
- **Accuracy is modest** (0.38-0.47 end-to-end on our lists) and below what larger systems report; the candidate filter alone caps it at 0.80-0.85.
- **Over-confident out of the box** (ECE about 0.19) and over-confident on some systematic mistakes (see the form test). Do not trigger irreversible actions (payments, deletions, sending messages) based on its confidence.
- **No element positions in the input**, full-page screenshots only (no viewport/scroll handling), pages truncated after 6400 px.
- **Distribution:** English web pages from the 2022-2023 Mind2Web snapshot; modern single-page apps, pop-ups, shadow DOM and iframes are not covered. Live DOM to tree conversion in `code/` is an approximation of the dataset format (the dataset has almost no state attributes).
- **One seed, one run.** No ablation isolates the effect of `always_pos`, the tree format, and rank 32 in the final configuration separately.
- **Possible contamination** with the pre-training data of the Gemma backbone (web pages) cannot be ruled out.

## How to use

Requirements: a GPU with bf16 support (inference memory was not measured; training peaked at 38.6 GB with gradients), `torch`, `transformers` (with Gemma 4 support), `peft`, `playwright`, `lxml`, `scikit-learn`, `pillow`. The code lives in `code/`; `inference_example.py` shows a complete loop.

```python
import sys; sys.path.insert(0, "code")
from m2w_form_demo import Gemma, aday_listesi, softmax

model = Gemma("senpaicq/m2w-gemma4-e2b-pointer", shots_dir="tmp_shots")  # downloads adapter + head

# `page` is a Playwright page (viewport width 1280). Candidates are extracted from the live DOM:
cands, raw = aday_listesi(page, task="Find flights from NYC to Paris", history=[], arm="etiketli")
png = page.screenshot(full_page=True)
ex = {"task": "Find flights from NYC to Paris", "history": [], "cands": cands, "pos_idx": [], "op": "CLICK"}
logits, op_logits = model(ex, png, None)     # logits has n_candidates + 1 entries; the last one is "none"
p = softmax(logits, T=1.0)                   # T = 1.0 is raw; fit your own temperature on held-out data
j = int(p.argmax())
print("none" if j == len(cands) else cands[j]["text"], float(p[j]))
```

Files in this repository: `adapter_model.safetensors` and `adapter_config.json` (LoRA), `head.pt` (pointer and operation heads), `m2w_config.json` (inference settings), `results/` (validation log, final metrics, per-step top-1 confidence and correctness for each split; no dataset content), `code/` (data preparation, filtering, formatting, training, evaluation and live-demo scripts), `inference_example.py`.

## Reproducing

The scripts in `code/` reproduce the pipeline: `m2w_prep.py` (candidates, TF-IDF order) -> `m2w_filter.py` (K = 160) -> `m2w_format.py --fmt tree` -> `m2w_ekran_goruntuleri_cikar.py` (screenshots) -> `m2w_train_lora_gorsel.py` (arguments used: `--layer 30 --epochs 4 --accum 8 --rank 32 --lora_mlp --max_cand_tok 64 --max_seq 4600 --img_tokens 280 --max_strips 5 --always_pos`). Comments are partly in Turkish. The dataset is not redistributed here; download it from its original source.

## License, data and attribution

- Our adapter weights and code are released under Apache-2.0. The base model `google/gemma-4-E2B` is listed as Apache-2.0 on its Hugging Face page; see its license terms.
- The model was trained on [Multimodal-Mind2Web](https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web). **We did not verify the dataset's license for this card; check the dataset page and terms before any commercial use.** No dataset content is included in this repository.
- Screenshots and DOM snapshots in the dataset come from third-party websites.

## Related work and credit

- **Kev** (Jared Palmer, 2026): open-source, Apache-2.0 family of small "System One" decision models that answer typed questions (choice, score, ...) with probabilities, built as a LoRA adapter plus a small pointer head on a Qwen base ([code](https://github.com/jaredpalmer/kev), [weights](https://huggingface.co/collections/jaredpalmer/kev)). The idea of a LoRA adapter with a pointer head over a closed set of options, and of reading calibrated probabilities from it, comes from this line of work. We are not affiliated with it, and this model is not a Kev checkpoint.
- **What is different here:** a different backbone (Gemma 4 E2B, a vision-language model), screenshots as input, a task-specific candidate set built from the DOM (hundreds of web elements per step), a learned "none" key and an operation head, LoRA on attention and MLP projections of layers 0-29 (rank 32), and an evaluation on Multimodal-Mind2Web with calibration and risk-coverage metrics. We did not compare against Kev checkpoints.
- **Benchmarks and data:** Mind2Web and Multimodal-Mind2Web (see Citation).

## Citation

If you use this model, please also cite the benchmark and the multimodal extension:

```bibtex
@inproceedings{deng2023mind2web,
  title     = {Mind2Web: Towards a Generalist Agent for the Web},
  author    = {Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Samuel and Wang, Boshi and Sun, Huan and Su, Yu},
  booktitle = {NeurIPS Datasets and Benchmarks Track},
  year      = {2023}
}
@inproceedings{zheng2024seeact,
  title     = {GPT-4V(ision) is a Generalist Web Agent, if Grounded},
  author    = {Zheng, Boyuan and Gou, Boyu and Kil, Jihyung and Sun, Huan and Su, Yu},
  booktitle = {ICML},
  year      = {2024}
}
```