decosa-commit-detector-xlmr-large

Before a browser or computer-use agent clicks an element, this model says whether the click commits something that is hard to undo, and what kind: pay, delete, send, publish, submit, security (passwords, 2FA, keys, permissions) or none. The agent, or a guard around it, stops and asks a person before any commit.

It reads the element's role and label plus the page title, URL path, heading and nearby text, so a generic "OK", "Continue" or "Weiter" is judged by the page it sits on. It covers 34 languages (36 language codes with pt-BR and es-419), including all 24 official EU languages. It runs on a CPU in about 50-80 ms per click; no LLM call.

Why: agents usually gate clicks with English keyword lists. Ours had 12 regexes. On the test sets below they caught under 1% of commits, because most commits are phrased in other languages or with words the list doesn't have.

Results

Two test sets, never trained on. Binary = commit vs none. False-commit rate = share of harmless clicks the model flags, i.e. how often a person is asked for no reason. The baselines saw the same fields.

Independent set (720 cases, 36 language codes, written by a different author from the label definitions only; the most honest test):

	Commit recall	False-commit rate	Binary accuracy	Kind accuracy
12 English keyword regexes	0.2%	0.7%	39.9%	—
Zero-shot Qwen3.8-27B (one-word answer, greedy)	98.1%	1.4%	98.3%	97.1%
This model (argmax)	98.4%	10.1%	95.0%	91.4%
This model at the shipped safety threshold	99.1%	13.2%	94.2%	90.6%

Held-out site kinds (3,206 cases from 8 kinds of website never seen in training, same generator as the training data):

	Commit recall	False-commit rate	Binary accuracy	Kind accuracy
12 English keyword regexes	0.6%	0.5%	37.9%	—
Zero-shot Qwen3.8-27B	93.2%	6.2%	93.4%	90.6%
This model (argmax)	97.2%	5.4%	96.2%	94.0%
This model at the shipped safety threshold	97.9%	6.9%	96.1%	93.6%

So: it catches commits as well as a 27B LLM does, far better than a keyword list, but on independently written cases it asks a person needlessly about 1 time in 10 (Qwen: 1 in 70). AUROC commit vs none: .989 (independent), .987 (held-out). By kind on the independent set: delete, security and send 72/72 caught, submit 71/72, pay and publish 69/72. Weakest languages there: Bulgarian and English 85% binary accuracy (20 cases each); Croatian, Italian, Maltese, Romanian 90%.

Speed on CPU (fp32, 8 threads, one click at a time): p50 77 ms, p95 88 ms on a busy server; 47 ms p50 on an idle 24-thread box.

All numbers are in eval_summary.json.

Intended use

• A pre-click guard for browser and computer-use agents: if commit is true, pause and ask a person, show the kind.
• Use it together with your own rules (ours: model OR keyword list), not instead of a permission system.

Not for: deciding on its own that an action is safe to take without a person, or any use where a missed commit is unacceptable without another safeguard. A none answer means the model didn't recognise a commit, not that there isn't one.

Limitations

• Synthetic training data. Real sites have longer and messier accessibility trees, icon-only buttons, cookie banners and dialogs. It has not yet been measured on real agent traces.
• More needless confirmations than an LLM (10% vs 1.4% of harmless clicks on the independent set). Typical false commits: buttons that only look final ("Continue" mid-flow, "Delete" on a search-filter chip).
• Text only: it doesn't see the screenshot. A button whose effect is only visible in an image is out of scope.
• Irish and Maltese training rows are machine translations, so those languages are less certain.
• The label set is a policy choice: "add to cart" is not a commit; cancelling a subscription counts as delete. Change the threshold or which kinds need approval to fit your policy.

Training

Base	FacebookAI/xlm-roberta-large @ c23d21b0 (560M parameters), MIT
Data	16,566 synthetic UI moments (element, page title, URL path, heading, nearby text, label) for 42 kinds of website, written natively in each language by Qwen3.8-27B; Irish and Maltese rows machine-translated from English by EuroLLM-9B-Instruct. The label is the kind the generator was asked to write; rows where it reported writing another kind were dropped. All sites, companies and people are invented. The data and generator are not released.
Method	Full fine-tune for sequence classification, 7 classes; AdamW (weight decay 0.01), lr 1.5e-5, cosine with 6% warm-up, batch 32, 3 epochs, bf16 autocast, label smoothing 0.05, max 160 tokens (element first); augmentation drops title / URL / context 12% each and changes the label's case 10%. 145 s on one H200.
Selection	Three bases were trained (XLM-R large, mmBERT-base, mmBERT-small); XLM-R large was picked on the dev split, before any test number was read. The safety threshold (p(none) < 0.70 means commit) was set on dev for ≥ 98% commit recall.

Usage

pip install transformers torch huggingface_hub
python usage.py '{"role":"button","name":"Weiter","heading":"Bestellung prüfen","title":"Kasse","url":"/checkout/review","context":"Gesamt 49,90 EUR, zahlungspflichtig"}'
# {"commit": true, "kind": "pay", "probs": {...}}

from usage import CommitDetector
cd = CommitDetector()                         # or CommitDetector(threshold_pnone=0.5) for fewer confirmations
cd([{"role": "button", "name": "...", "heading": "...", "title": "...", "url": "...", "context": "..."}])

The input string must be built exactly as usage.text_of() does (role: name | heading: ... | title: ... | url: ... | context: ..., empty fields left out). decosa_commit.json holds the class order, the max length and the threshold.

Files: model.safetensors (fp32), config.json, tokenizer files, decosa_commit.json, usage.py, eval_summary.json, SHA256SUMS, LICENSE, NOTICE.

Licence

Apache-2.0 for the weights and code in this repository. The base model is MIT (Facebook, Inc. and its affiliates); its licence notice is in NOTICE.