folate / README.md
asermnasr's picture
Update README.md
b02fd60 verified
|
Raw History Blame Contribute Delete
10.4 kB
---
license: apache-2.0
language:
- en
base_model:
- google/electra-base-discriminator
datasets:
- osunlp/Mind2Web
pipeline_tag: zero-shot-classification
library_name: onnxruntime
model-index:
- name: folate
results:
- task:
type: zero-shot-classification
name: DOM Element Grounding
dataset:
name: Mind2Web (test_task)
type: osunlp/Mind2Web
split: test_task
metrics:
- type: accuracy
value: 93.31
name: Top-1 Accuracy
- type: mrr
value: 0.9566
name: MRR
- task:
type: zero-shot-classification
name: DOM Element Grounding
dataset:
name: Mind2Web (test_website)
type: osunlp/Mind2Web
split: test_website
metrics:
- type: accuracy
value: 88.9
name: Top-1 Accuracy
- type: mrr
value: 0.9274
name: MRR
- task:
type: zero-shot-classification
name: DOM Element Grounding
dataset:
name: Mind2Web (test_domain)
type: osunlp/Mind2Web
split: test_domain
metrics:
- type: accuracy
value: 92.08
name: Top-1 Accuracy
- type: mrr
value: 0.9491
name: MRR
tags:
- onnx
- int8
- dom
- web-agent
- browser-automation
- cross-encoder
- computer-use
---
<p align="center">
<img src="https://raw.githubusercontent.com/figranium/folate/main/banner.png" alt="Folate" style="max-height: 220px; object-fit: cover;">
</p>
<p align="center">
<a href="https://huggingface.co/spaces/figranium/folate-demo"><strong>Demo</strong></a>
&nbsp;&bull;&nbsp;
<a href="https://pypi.org/project/folate/"><strong>PyPI</strong></a>
&nbsp;&bull;&nbsp;
<a href="https://github.com/figranium/folate"><strong>GitHub</strong></a>
&nbsp;&bull;&nbsp;
<a href="https://github.com/figranium/folate/blob/main/LICENSE"><strong>Apache 2.0 License</strong></a>
</p>
**Folate** is a lightweight, CPU-efficient neural DOM element ranker built for browser automation within tight memory constraints (<2 GB total footprint alongside a headless browser). Fine-tuned from [`google/electra-base-discriminator`](https://huggingface.co/google/electra-base-discriminator) as a discriminative cross-encoder on [`osunlp/Mind2Web`](https://huggingface.co/datasets/osunlp/Mind2Web) and dynamically quantized to **INT8 ONNX (~107 MB)**, Folate runs purely via `onnxruntime` and `tokenizers`—requiring no PyTorch or `transformers` at inference time.
Folate is designed to act as a fast, zero-hallucination UI grounding engine for single-step browser actions (`click`, `type`, `select`, `hover`) on resource-constrained laptops, Apple Silicon devices, and CPU-only containers.
| Artifact | Backbone | Precision / Format | Disk Size | Runtime RAM |
| :--- | :--- | :--- | :--- | :--- |
| **`model_quantized.onnx` (Default)** | `google/electra-base-discriminator` | Dynamic `QUInt8` ONNX | ~107 MB | ~180–250 MB |
| **`folate-base` (Checkpoint)** | `google/electra-base-discriminator` | `float32` Safetensors | ~418 MB | ~1.4+ GB (PyTorch) |
### Inputs and outputs
* **Input:**
* A natural-language action query (e.g., `"click Search Flights button"` or `"type product name in Amazon search box"`).
* Candidate interactive DOM elements extracted from raw HTML via `lxml` XPath (`<a>`, `<button>`, `<input>`, `<select>`, `<textarea>`, ARIA interactive roles, and `onclick`/`tabindex` elements) and serialized into a structured `96`-token string:
```text
[TAG] button [TYPE] submit [DATA-TESTID] search-flights [TEXT] Search Flights [PARENT] form
```
* **Output:**
* A calibrated relevance probability (`0.0` to `1.0`, derived via sigmoid over a single sequence-classification logit) and a resolved CSS or XPath selector guaranteed to exist in the source DOM, ranked in descending order of relevance.
### Citation
```bibtex
@misc{folate_2026,
title = {Folate: Lightweight CPU-Efficient Neural DOM Element Ranker},
author = {Figranium},
year = {2026},
url = {https://huggingface.co/figranium/folate}
}
```
---
## Model Data
Data used for model training and how the data was processed.
### Training Dataset
Folate was fine-tuned on the training split of [`osunlp/Mind2Web`](https://huggingface.co/datasets/osunlp/Mind2Web), a generalist web-agent dataset spanning real-world websites across travel, shopping, entertainment, and information workflows.
### Data Preprocessing
Each Mind2Web action step was streamed and converted into binary `(query, serialized_element)` classification pairs using `scripts/prepare_data.py`:
* **Query Normalization:** Action descriptions were stripped of bracketed syntactic markers and normalized to concise natural-language instructions.
* **Structured DOM Serialization:** Candidate elements were parsed with `lxml.html` and serialized using their lowercase tag name, up to 10 high-signal HTML attributes (`id`, `name`, `type`, `role`, `aria-label`, `placeholder`, `title`, `alt`, `href`, `data-testid`), normalized inner text (capped at `80` characters), and immediate `[PARENT]` tag.
* **Negative Sampling & Splitting:** Each ground-truth positive element was paired with up to `7` sampled negative candidates from the same page state and split `90% / 10%` into training and validation sets.
---
## Implementation Information
Details about the model internals and runtime optimization.
* **Architecture:** `ElectraForSequenceClassification` (`12` transformer layers, `768` hidden size, `12` attention heads, `num_labels=1`) initialized from `google/electra-base-discriminator`.
* **Sequence Length:** Truncated and padded to a maximum length of `96` WordPiece tokens (`vocab_size = 30,522`).
* **Quantization & Export:** Exported from PyTorch to FP32 ONNX (`opset_version=17`) with dynamic batch and sequence axes, then quantized using `onnxruntime.quantization.quantize_dynamic` (`QuantType.QUInt8`).
* **Inference Pipeline:** Uses a two-stage CPU retrieval pipeline in `FolateSelectorFinder`—fast lexical pre-filtering (`max_candidates=48`) followed by cache-friendly mini-batching (`batch_size=16`) on `CPUExecutionProvider`.
---
## Evaluation & Benchmarks
Folate is evaluated on the three standard generalization splits of **Mind2Web** (`test_task`, `test_website`, and `test_domain`) across **6,037 action steps** on real-world webpages.
### Mind2Web Results (INT8 ONNX · CPU)
| Split | Description | Steps | Avg Candidates | Top-1 | Top-3 | Top-5 | MRR | Avg Latency (2 vCPU) |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **`test_task`** | Unseen tasks on seen websites | 1,256 | 50.0 | **93.31%** | 97.61% | 98.33% | 0.9566 | 2,309.9 ms/step |
| **`test_website`** | Completely unseen websites | 955 | 50.0 | **88.90%** | 95.92% | 96.65% | 0.9274 | 4,328.2 ms/step |
| **`test_domain`** | Unseen domains & verticals | 3,826 | 49.8 | **92.08%** | 97.28% | 98.61% | 0.9491 | 2,431.0 ms/step |
### Methodology & Setup
- **Task Formulation:** At each step, the model receives the natural-language task instruction, action history context, and up to **50 candidate DOM elements** (`Avg Cands ≈ 50.0`) extracted from the page snapshot, and outputs a relevance logit per candidate.
- **Metrics:**
- **Top-1 / Top-3 / Top-5:** Percentage of steps where a ground-truth target node (`pos_candidates`) appears within the top ranked elements.
- **MRR (Mean Reciprocal Rank):** Average for the highest-ranked ground-truth element across all evaluated steps.
- **Hardware & Runtime:** Latency is measured end-to-end per step on **2 vCPUs** with **no GPU acceleration**, using the exported INT8 quantized ONNX artifact (`onnxruntime`). Variance between `test_website` (`4,328.2 ms/step`) and `test_domain` (`2,431.0 ms/step`) reflects longer tokenized DOM attribute sequences on unseen site templates rather than candidate count.
---
## Ethics and Safety
Evaluation approach and results.
### Evaluation Approach
Folate is a discriminative element ranker rather than a generative text or code model; it does not generate free-form text, execute arbitrary code, or store user credentials. Evaluations focused on selector grounding fidelity (verifying that returned selectors resolve strictly to existing interactive elements in the provided HTML) and failure-mode analysis on ambiguous sibling elements.
### Evaluation Results
Across tested DOM snapshots, 100% of returned selectors mapped to valid nodes in the input HTML tree with zero fabricated attributes or hallucinated IDs.
---
## Usage and Limitations
Instructions on how to use the model as well as its limitations.
### Usage
Install the lightweight inference runtime from [`github.com/figranium/folate`](https://github.com/figranium/folate):
```bash
pip install folate
```
Rank interactive DOM elements against a natural-language action query (`FolateSelectorFinder` automatically downloads and caches `figranium/folate` on first run):
```python
from folate import FolateSelectorFinder
finder = FolateSelectorFinder()
matches = finder.rank("click Search Flights button", html_source, top_k=3)
for match in matches:
print(f"{match['score']:.4f} | {match['selector']} | {match['element']}")
```
To pre-download the quantized model weights locally:
```bash
hf download figranium/folate --local-dir ./folate-int8
```
### System Requirements
* **Disk Space:** ~108 MB (`model_quantized.onnx` + `tokenizer.json`)
* **Memory (RAM):** ~180–250 MB during CPU inference (fits within a <2 GB total system budget alongside headless Chromium)
* **Hardware:** Standard x86_64 or ARM64 CPU; no GPU required
### Limitations
* **Single-Step Grounding Only:** Folate ranks candidate elements for a single action query (`click`, `type`, `select`, `hover`); it does not perform multi-step workflow planning, state tracking, or unstructured text extraction.
* **Local Structural Context:** Elements are serialized using their own attributes, inner text, and immediate `[PARENT]` tag—identical sibling elements lacking distinguishing attributes or text (e.g., repeated `"Add to Cart"` buttons in a grid) cannot be disambiguated spatially.
* **Confidence Calibration:** Because training uses a `1:7` positive-to-negative ratio, well-calibrated top-1 positive scores typically fall in the `0.58–0.85` range rather than `0.99`. For ambiguous pages (`score < 0.55`), pair Folate with a higher-level planner over the top-5 candidates.