File size: 10,390 Bytes
73d25ea
 
5adc0ea
 
 
 
 
 
4dd4661
5adc0ea
02a625d
62c88f2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5adc0ea
 
 
 
 
 
 
62c88f2
73d25ea
5adc0ea
09dc6d1
 
 
 
 
d1ffc1d
09dc6d1
909853f
 
09dc6d1
 
909853f
09dc6d1
5adc0ea
88a5f12
5adc0ea
88a5f12
5adc0ea
88a5f12
 
 
 
5adc0ea
88a5f12
5adc0ea
88a5f12
 
 
 
 
 
 
 
5adc0ea
88a5f12
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
02a625d
88a5f12
02a625d
88a5f12
02a625d
88a5f12
02a625d
 
 
 
 
88a5f12
02a625d
88a5f12
02a625d
 
b02fd60
02a625d
 
 
88a5f12
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5adc0ea
 
8e26566
88a5f12
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ee22df8
88a5f12
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
---
license: apache-2.0
language:
- en
base_model:
- google/electra-base-discriminator
datasets:
- osunlp/Mind2Web
pipeline_tag: zero-shot-classification
library_name: onnxruntime
model-index:
- name: folate
  results:
  - task:
      type: zero-shot-classification
      name: DOM Element Grounding
    dataset:
      name: Mind2Web (test_task)
      type: osunlp/Mind2Web
      split: test_task
    metrics:
    - type: accuracy
      value: 93.31
      name: Top-1 Accuracy
    - type: mrr
      value: 0.9566
      name: MRR
  - task:
      type: zero-shot-classification
      name: DOM Element Grounding
    dataset:
      name: Mind2Web (test_website)
      type: osunlp/Mind2Web
      split: test_website
    metrics:
    - type: accuracy
      value: 88.9
      name: Top-1 Accuracy
    - type: mrr
      value: 0.9274
      name: MRR
  - task:
      type: zero-shot-classification
      name: DOM Element Grounding
    dataset:
      name: Mind2Web (test_domain)
      type: osunlp/Mind2Web
      split: test_domain
    metrics:
    - type: accuracy
      value: 92.08
      name: Top-1 Accuracy
    - type: mrr
      value: 0.9491
      name: MRR
tags:
- onnx
- int8
- dom
- web-agent
- browser-automation
- cross-encoder
- computer-use
---

<p align="center">
  <img src="https://raw.githubusercontent.com/figranium/folate/main/banner.png" alt="Folate" style="max-height: 220px; object-fit: cover;">
</p>

<p align="center">
  <a href="https://huggingface.co/spaces/figranium/folate-demo"><strong>Demo</strong></a>
  &nbsp;&bull;&nbsp;
  <a href="https://pypi.org/project/folate/"><strong>PyPI</strong></a>
  &nbsp;&bull;&nbsp;
  <a href="https://github.com/figranium/folate"><strong>GitHub</strong></a>
  &nbsp;&bull;&nbsp;
  <a href="https://github.com/figranium/folate/blob/main/LICENSE"><strong>Apache 2.0 License</strong></a>
</p>

**Folate** is a lightweight, CPU-efficient neural DOM element ranker built for browser automation within tight memory constraints (<2 GB total footprint alongside a headless browser). Fine-tuned from [`google/electra-base-discriminator`](https://huggingface.co/google/electra-base-discriminator) as a discriminative cross-encoder on [`osunlp/Mind2Web`](https://huggingface.co/datasets/osunlp/Mind2Web) and dynamically quantized to **INT8 ONNX (~107 MB)**, Folate runs purely via `onnxruntime` and `tokenizers`—requiring no PyTorch or `transformers` at inference time.

Folate is designed to act as a fast, zero-hallucination UI grounding engine for single-step browser actions (`click`, `type`, `select`, `hover`) on resource-constrained laptops, Apple Silicon devices, and CPU-only containers.

| Artifact | Backbone | Precision / Format | Disk Size | Runtime RAM |
| :--- | :--- | :--- | :--- | :--- |
| **`model_quantized.onnx` (Default)** | `google/electra-base-discriminator` | Dynamic `QUInt8` ONNX | ~107 MB | ~180–250 MB |
| **`folate-base` (Checkpoint)** | `google/electra-base-discriminator` | `float32` Safetensors | ~418 MB | ~1.4+ GB (PyTorch) |

### Inputs and outputs

* **Input:**
  * A natural-language action query (e.g., `"click Search Flights button"` or `"type product name in Amazon search box"`).
  * Candidate interactive DOM elements extracted from raw HTML via `lxml` XPath (`<a>`, `<button>`, `<input>`, `<select>`, `<textarea>`, ARIA interactive roles, and `onclick`/`tabindex` elements) and serialized into a structured `96`-token string:
    ```text
    [TAG] button [TYPE] submit [DATA-TESTID] search-flights [TEXT] Search Flights [PARENT] form
    ```
* **Output:**
  * A calibrated relevance probability (`0.0` to `1.0`, derived via sigmoid over a single sequence-classification logit) and a resolved CSS or XPath selector guaranteed to exist in the source DOM, ranked in descending order of relevance.

### Citation

```bibtex
@misc{folate_2026,
  title  = {Folate: Lightweight CPU-Efficient Neural DOM Element Ranker},
  author = {Figranium},
  year   = {2026},
  url    = {https://huggingface.co/figranium/folate}
}
```

---

## Model Data

Data used for model training and how the data was processed.

### Training Dataset

Folate was fine-tuned on the training split of [`osunlp/Mind2Web`](https://huggingface.co/datasets/osunlp/Mind2Web), a generalist web-agent dataset spanning real-world websites across travel, shopping, entertainment, and information workflows.

### Data Preprocessing

Each Mind2Web action step was streamed and converted into binary `(query, serialized_element)` classification pairs using `scripts/prepare_data.py`:

* **Query Normalization:** Action descriptions were stripped of bracketed syntactic markers and normalized to concise natural-language instructions.
* **Structured DOM Serialization:** Candidate elements were parsed with `lxml.html` and serialized using their lowercase tag name, up to 10 high-signal HTML attributes (`id`, `name`, `type`, `role`, `aria-label`, `placeholder`, `title`, `alt`, `href`, `data-testid`), normalized inner text (capped at `80` characters), and immediate `[PARENT]` tag.
* **Negative Sampling & Splitting:** Each ground-truth positive element was paired with up to `7` sampled negative candidates from the same page state and split `90% / 10%` into training and validation sets.

---

## Implementation Information

Details about the model internals and runtime optimization.

* **Architecture:** `ElectraForSequenceClassification` (`12` transformer layers, `768` hidden size, `12` attention heads, `num_labels=1`) initialized from `google/electra-base-discriminator`.
* **Sequence Length:** Truncated and padded to a maximum length of `96` WordPiece tokens (`vocab_size = 30,522`).
* **Quantization & Export:** Exported from PyTorch to FP32 ONNX (`opset_version=17`) with dynamic batch and sequence axes, then quantized using `onnxruntime.quantization.quantize_dynamic` (`QuantType.QUInt8`).
* **Inference Pipeline:** Uses a two-stage CPU retrieval pipeline in `FolateSelectorFinder`—fast lexical pre-filtering (`max_candidates=48`) followed by cache-friendly mini-batching (`batch_size=16`) on `CPUExecutionProvider`.

---

## Evaluation & Benchmarks

Folate is evaluated on the three standard generalization splits of **Mind2Web** (`test_task`, `test_website`, and `test_domain`) across **6,037 action steps** on real-world webpages. 

### Mind2Web Results (INT8 ONNX · CPU)

| Split | Description | Steps | Avg Candidates | Top-1 | Top-3 | Top-5 | MRR | Avg Latency (2 vCPU) |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **`test_task`** | Unseen tasks on seen websites | 1,256 | 50.0 | **93.31%** | 97.61% | 98.33% | 0.9566 | 2,309.9 ms/step |
| **`test_website`** | Completely unseen websites | 955 | 50.0 | **88.90%** | 95.92% | 96.65% | 0.9274 | 4,328.2 ms/step |
| **`test_domain`** | Unseen domains & verticals | 3,826 | 49.8 | **92.08%** | 97.28% | 98.61% | 0.9491 | 2,431.0 ms/step |

### Methodology & Setup

- **Task Formulation:** At each step, the model receives the natural-language task instruction, action history context, and up to **50 candidate DOM elements** (`Avg Cands ≈ 50.0`) extracted from the page snapshot, and outputs a relevance logit per candidate.
- **Metrics:**
  - **Top-1 / Top-3 / Top-5:** Percentage of steps where a ground-truth target node (`pos_candidates`) appears within the top ranked elements.
  - **MRR (Mean Reciprocal Rank):** Average for the highest-ranked ground-truth element across all evaluated steps.
- **Hardware & Runtime:** Latency is measured end-to-end per step on **2 vCPUs** with **no GPU acceleration**, using the exported INT8 quantized ONNX artifact (`onnxruntime`). Variance between `test_website` (`4,328.2 ms/step`) and `test_domain` (`2,431.0 ms/step`) reflects longer tokenized DOM attribute sequences on unseen site templates rather than candidate count.
  
---

## Ethics and Safety

Evaluation approach and results.

### Evaluation Approach

Folate is a discriminative element ranker rather than a generative text or code model; it does not generate free-form text, execute arbitrary code, or store user credentials. Evaluations focused on selector grounding fidelity (verifying that returned selectors resolve strictly to existing interactive elements in the provided HTML) and failure-mode analysis on ambiguous sibling elements.

### Evaluation Results

Across tested DOM snapshots, 100% of returned selectors mapped to valid nodes in the input HTML tree with zero fabricated attributes or hallucinated IDs.

---

## Usage and Limitations

Instructions on how to use the model as well as its limitations.

### Usage

Install the lightweight inference runtime from [`github.com/figranium/folate`](https://github.com/figranium/folate):

```bash
pip install folate
```

Rank interactive DOM elements against a natural-language action query (`FolateSelectorFinder` automatically downloads and caches `figranium/folate` on first run):

```python
from folate import FolateSelectorFinder

finder = FolateSelectorFinder()
matches = finder.rank("click Search Flights button", html_source, top_k=3)

for match in matches:
    print(f"{match['score']:.4f} | {match['selector']} | {match['element']}")
```

To pre-download the quantized model weights locally:

```bash
hf download figranium/folate --local-dir ./folate-int8
```

### System Requirements

* **Disk Space:** ~108 MB (`model_quantized.onnx` + `tokenizer.json`)
* **Memory (RAM):** ~180–250 MB during CPU inference (fits within a <2 GB total system budget alongside headless Chromium)
* **Hardware:** Standard x86_64 or ARM64 CPU; no GPU required

### Limitations

* **Single-Step Grounding Only:** Folate ranks candidate elements for a single action query (`click`, `type`, `select`, `hover`); it does not perform multi-step workflow planning, state tracking, or unstructured text extraction.
* **Local Structural Context:** Elements are serialized using their own attributes, inner text, and immediate `[PARENT]` tag—identical sibling elements lacking distinguishing attributes or text (e.g., repeated `"Add to Cart"` buttons in a grid) cannot be disambiguated spatially.
* **Confidence Calibration:** Because training uses a `1:7` positive-to-negative ratio, well-calibrated top-1 positive scores typically fall in the `0.58–0.85` range rather than `0.99`. For ambiguous pages (`score < 0.55`), pair Folate with a higher-level planner over the top-5 candidates.