|
Download README.md from figranium/folate: direct link, hf CLI and curl.
- Browser
- Download file 10.4 kB
-
https://huggingface.co/figranium/folate/resolve/main/README.md
- Command line
-
hf download hf://figranium/folate/README.md
-
curl -L -o README.md https://huggingface.co/figranium/folate/resolve/main/README.md
10.4 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| base_model: | |
| - google/electra-base-discriminator | |
| datasets: | |
| - osunlp/Mind2Web | |
| pipeline_tag: zero-shot-classification | |
| library_name: onnxruntime | |
| model-index: | |
| - name: folate | |
| results: | |
| - task: | |
| type: zero-shot-classification | |
| name: DOM Element Grounding | |
| dataset: | |
| name: Mind2Web (test_task) | |
| type: osunlp/Mind2Web | |
| split: test_task | |
| metrics: | |
| - type: accuracy | |
| value: 93.31 | |
| name: Top-1 Accuracy | |
| - type: mrr | |
| value: 0.9566 | |
| name: MRR | |
| - task: | |
| type: zero-shot-classification | |
| name: DOM Element Grounding | |
| dataset: | |
| name: Mind2Web (test_website) | |
| type: osunlp/Mind2Web | |
| split: test_website | |
| metrics: | |
| - type: accuracy | |
| value: 88.9 | |
| name: Top-1 Accuracy | |
| - type: mrr | |
| value: 0.9274 | |
| name: MRR | |
| - task: | |
| type: zero-shot-classification | |
| name: DOM Element Grounding | |
| dataset: | |
| name: Mind2Web (test_domain) | |
| type: osunlp/Mind2Web | |
| split: test_domain | |
| metrics: | |
| - type: accuracy | |
| value: 92.08 | |
| name: Top-1 Accuracy | |
| - type: mrr | |
| value: 0.9491 | |
| name: MRR | |
| tags: | |
| - onnx | |
| - int8 | |
| - dom | |
| - web-agent | |
| - browser-automation | |
| - cross-encoder | |
| - computer-use | |
| <p align="center"> | |
| <img src="https://raw.githubusercontent.com/figranium/folate/main/banner.png" alt="Folate" style="max-height: 220px; object-fit: cover;"> | |
| </p> | |
| <p align="center"> | |
| <a href="https://huggingface.co/spaces/figranium/folate-demo"><strong>Demo</strong></a> | |
| • | |
| <a href="https://pypi.org/project/folate/"><strong>PyPI</strong></a> | |
| • | |
| <a href="https://github.com/figranium/folate"><strong>GitHub</strong></a> | |
| • | |
| <a href="https://github.com/figranium/folate/blob/main/LICENSE"><strong>Apache 2.0 License</strong></a> | |
| </p> | |
| **Folate** is a lightweight, CPU-efficient neural DOM element ranker built for browser automation within tight memory constraints (<2 GB total footprint alongside a headless browser). Fine-tuned from [`google/electra-base-discriminator`](https://huggingface.co/google/electra-base-discriminator) as a discriminative cross-encoder on [`osunlp/Mind2Web`](https://huggingface.co/datasets/osunlp/Mind2Web) and dynamically quantized to **INT8 ONNX (~107 MB)**, Folate runs purely via `onnxruntime` and `tokenizers`—requiring no PyTorch or `transformers` at inference time. | |
| Folate is designed to act as a fast, zero-hallucination UI grounding engine for single-step browser actions (`click`, `type`, `select`, `hover`) on resource-constrained laptops, Apple Silicon devices, and CPU-only containers. | |
| | Artifact | Backbone | Precision / Format | Disk Size | Runtime RAM | | |
| | :--- | :--- | :--- | :--- | :--- | | |
| | **`model_quantized.onnx` (Default)** | `google/electra-base-discriminator` | Dynamic `QUInt8` ONNX | ~107 MB | ~180–250 MB | | |
| | **`folate-base` (Checkpoint)** | `google/electra-base-discriminator` | `float32` Safetensors | ~418 MB | ~1.4+ GB (PyTorch) | | |
| ### Inputs and outputs | |
| * **Input:** | |
| * A natural-language action query (e.g., `"click Search Flights button"` or `"type product name in Amazon search box"`). | |
| * Candidate interactive DOM elements extracted from raw HTML via `lxml` XPath (`<a>`, `<button>`, `<input>`, `<select>`, `<textarea>`, ARIA interactive roles, and `onclick`/`tabindex` elements) and serialized into a structured `96`-token string: | |
| ```text | |
| [TAG] button [TYPE] submit [DATA-TESTID] search-flights [TEXT] Search Flights [PARENT] form | |
| ``` | |
| * **Output:** | |
| * A calibrated relevance probability (`0.0` to `1.0`, derived via sigmoid over a single sequence-classification logit) and a resolved CSS or XPath selector guaranteed to exist in the source DOM, ranked in descending order of relevance. | |
| ### Citation | |
| ```bibtex | |
| @misc{folate_2026, | |
| title = {Folate: Lightweight CPU-Efficient Neural DOM Element Ranker}, | |
| author = {Figranium}, | |
| year = {2026}, | |
| url = {https://huggingface.co/figranium/folate} | |
| } | |
| ``` | |
| --- | |
| ## Model Data | |
| Data used for model training and how the data was processed. | |
| ### Training Dataset | |
| Folate was fine-tuned on the training split of [`osunlp/Mind2Web`](https://huggingface.co/datasets/osunlp/Mind2Web), a generalist web-agent dataset spanning real-world websites across travel, shopping, entertainment, and information workflows. | |
| ### Data Preprocessing | |
| Each Mind2Web action step was streamed and converted into binary `(query, serialized_element)` classification pairs using `scripts/prepare_data.py`: | |
| * **Query Normalization:** Action descriptions were stripped of bracketed syntactic markers and normalized to concise natural-language instructions. | |
| * **Structured DOM Serialization:** Candidate elements were parsed with `lxml.html` and serialized using their lowercase tag name, up to 10 high-signal HTML attributes (`id`, `name`, `type`, `role`, `aria-label`, `placeholder`, `title`, `alt`, `href`, `data-testid`), normalized inner text (capped at `80` characters), and immediate `[PARENT]` tag. | |
| * **Negative Sampling & Splitting:** Each ground-truth positive element was paired with up to `7` sampled negative candidates from the same page state and split `90% / 10%` into training and validation sets. | |
| --- | |
| ## Implementation Information | |
| Details about the model internals and runtime optimization. | |
| * **Architecture:** `ElectraForSequenceClassification` (`12` transformer layers, `768` hidden size, `12` attention heads, `num_labels=1`) initialized from `google/electra-base-discriminator`. | |
| * **Sequence Length:** Truncated and padded to a maximum length of `96` WordPiece tokens (`vocab_size = 30,522`). | |
| * **Quantization & Export:** Exported from PyTorch to FP32 ONNX (`opset_version=17`) with dynamic batch and sequence axes, then quantized using `onnxruntime.quantization.quantize_dynamic` (`QuantType.QUInt8`). | |
| * **Inference Pipeline:** Uses a two-stage CPU retrieval pipeline in `FolateSelectorFinder`—fast lexical pre-filtering (`max_candidates=48`) followed by cache-friendly mini-batching (`batch_size=16`) on `CPUExecutionProvider`. | |
| --- | |
| ## Evaluation & Benchmarks | |
| Folate is evaluated on the three standard generalization splits of **Mind2Web** (`test_task`, `test_website`, and `test_domain`) across **6,037 action steps** on real-world webpages. | |
| ### Mind2Web Results (INT8 ONNX · CPU) | |
| | Split | Description | Steps | Avg Candidates | Top-1 | Top-3 | Top-5 | MRR | Avg Latency (2 vCPU) | | |
| | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | **`test_task`** | Unseen tasks on seen websites | 1,256 | 50.0 | **93.31%** | 97.61% | 98.33% | 0.9566 | 2,309.9 ms/step | | |
| | **`test_website`** | Completely unseen websites | 955 | 50.0 | **88.90%** | 95.92% | 96.65% | 0.9274 | 4,328.2 ms/step | | |
| | **`test_domain`** | Unseen domains & verticals | 3,826 | 49.8 | **92.08%** | 97.28% | 98.61% | 0.9491 | 2,431.0 ms/step | | |
| ### Methodology & Setup | |
| - **Task Formulation:** At each step, the model receives the natural-language task instruction, action history context, and up to **50 candidate DOM elements** (`Avg Cands ≈ 50.0`) extracted from the page snapshot, and outputs a relevance logit per candidate. | |
| - **Metrics:** | |
| - **Top-1 / Top-3 / Top-5:** Percentage of steps where a ground-truth target node (`pos_candidates`) appears within the top ranked elements. | |
| - **MRR (Mean Reciprocal Rank):** Average for the highest-ranked ground-truth element across all evaluated steps. | |
| - **Hardware & Runtime:** Latency is measured end-to-end per step on **2 vCPUs** with **no GPU acceleration**, using the exported INT8 quantized ONNX artifact (`onnxruntime`). Variance between `test_website` (`4,328.2 ms/step`) and `test_domain` (`2,431.0 ms/step`) reflects longer tokenized DOM attribute sequences on unseen site templates rather than candidate count. | |
| --- | |
| ## Ethics and Safety | |
| Evaluation approach and results. | |
| ### Evaluation Approach | |
| Folate is a discriminative element ranker rather than a generative text or code model; it does not generate free-form text, execute arbitrary code, or store user credentials. Evaluations focused on selector grounding fidelity (verifying that returned selectors resolve strictly to existing interactive elements in the provided HTML) and failure-mode analysis on ambiguous sibling elements. | |
| ### Evaluation Results | |
| Across tested DOM snapshots, 100% of returned selectors mapped to valid nodes in the input HTML tree with zero fabricated attributes or hallucinated IDs. | |
| --- | |
| ## Usage and Limitations | |
| Instructions on how to use the model as well as its limitations. | |
| ### Usage | |
| Install the lightweight inference runtime from [`github.com/figranium/folate`](https://github.com/figranium/folate): | |
| ```bash | |
| pip install folate | |
| ``` | |
| Rank interactive DOM elements against a natural-language action query (`FolateSelectorFinder` automatically downloads and caches `figranium/folate` on first run): | |
| ```python | |
| from folate import FolateSelectorFinder | |
| finder = FolateSelectorFinder() | |
| matches = finder.rank("click Search Flights button", html_source, top_k=3) | |
| for match in matches: | |
| print(f"{match['score']:.4f} | {match['selector']} | {match['element']}") | |
| ``` | |
| To pre-download the quantized model weights locally: | |
| ```bash | |
| hf download figranium/folate --local-dir ./folate-int8 | |
| ``` | |
| ### System Requirements | |
| * **Disk Space:** ~108 MB (`model_quantized.onnx` + `tokenizer.json`) | |
| * **Memory (RAM):** ~180–250 MB during CPU inference (fits within a <2 GB total system budget alongside headless Chromium) | |
| * **Hardware:** Standard x86_64 or ARM64 CPU; no GPU required | |
| ### Limitations | |
| * **Single-Step Grounding Only:** Folate ranks candidate elements for a single action query (`click`, `type`, `select`, `hover`); it does not perform multi-step workflow planning, state tracking, or unstructured text extraction. | |
| * **Local Structural Context:** Elements are serialized using their own attributes, inner text, and immediate `[PARENT]` tag—identical sibling elements lacking distinguishing attributes or text (e.g., repeated `"Add to Cart"` buttons in a grid) cannot be disambiguated spatially. | |
| * **Confidence Calibration:** Because training uses a `1:7` positive-to-negative ratio, well-calibrated top-1 positive scores typically fall in the `0.58–0.85` range rather than `0.99`. For ambiguous pages (`score < 0.55`), pair Folate with a higher-level planner over the top-5 candidates. |