File size: 9,038 Bytes
460678d 5aa3d92 66c872e 5aa3d92 460678d 5aa3d92 66c872e 1d8d779 5aa3d92 66c872e 5aa3d92 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 | ---
license: apache-2.0
language:
- en
base_model:
- microsoft/Florence-2-base-ft
datasets:
- inclusionAI/VenusBench-CAPTCHA
- OpenCaptchaWorld/Open_CaptchaWorld
- hammer888/captcha-data
pipeline_tag: image-text-to-text
library_name: onnxruntime
model-index:
- name: fiptcha
results:
- task:
type: image-text-to-text
name: CAPTCHA Visual Grounding
dataset:
name: VenusBench-CAPTCHA (val.jsonl, 39 grounding samples)
type: inclusionAI/VenusBench-CAPTCHA
split: validation
metrics:
- type: accuracy
value: 79.5
name: IoU@0.5
- type: iou
value: 0.669
name: Mean IoU
tags:
- onnx
- int8
- florence2
- captcha
- vision-language
- browser-automation
- computer-use
---
<p align="center">
<img src="https://raw.githubusercontent.com/figranium/fiptcha/main/banner.png" alt="Fiptcha" style="max-height: 220px; object-fit: cover;">
</p>
<p align="center">
<a href="https://www.npmjs.com/package/fiptcha"><strong>npm</strong></a>
•
<a href="https://github.com/figranium/fiptcha"><strong>GitHub</strong></a>
•
<a href="https://github.com/figranium/fiptcha/blob/main/LICENSE"><strong>Apache 2.0 License</strong></a>
</p>
**Fiptcha** is a lightweight, CPU-efficient vision-language CAPTCHA solver built for browser automation within tight memory constraints. Fine-tuned from [`microsoft/Florence-2-base-ft`](https://huggingface.co/microsoft/Florence-2-base-ft) on CAPTCHA challenge data derived from [`inclusionAI/VenusBench-CAPTCHA`](https://huggingface.co/datasets/inclusionAI/VenusBench-CAPTCHA) and dynamically quantized to **INT8 ONNX (~431 MB total across 5 submodels)**, Fiptcha runs purely via `onnxruntime` and `transformers` / `tokenizers`—requiring no GPU or PyTorch at inference time.
Fiptcha is designed to act as a local CAPTCHA solving engine (`<OCR>` and `<CAPTION_TO_PHRASE_GROUNDING>` for text, icon/text click, slider puzzle, drag-end, image rotation, and visual reasoning challenges) on resource-constrained laptops, Apple Silicon devices, and CPU-only containers.
| Artifact | Backbone | Precision / Format | Disk Size |
| :--- | :--- | :--- | :--- |
| **ONNX INT8 Submodels (Default)** | `microsoft/Florence-2-base-ft` | Dynamic per-channel `QInt8` ONNX | ~431 MB (total) |
| **`fiptcha-florence2-merged` (Checkpoint)** | `microsoft/Florence-2-base-ft` | `float32` Safetensors | ~883.4 MB |
### Submodel Breakdown
| File | INT8 Size | Role |
| :--- | :---: | :--- |
| `vision_encoder.onnx` | 90 MB | DaViT vision backbone + 2D/temporal embeddings + projection |
| `embed_tokens.onnx` | 38 MB | Shared token embedding lookup |
| `encoder_model.onnx` | 42 MB | Multimodal BART-style 6-layer encoder |
| `decoder_model.onnx` | 131 MB | Autoregressive 6-layer decoder (initial step) |
| `decoder_with_past_model.onnx` | 124 MB | Autoregressive 6-layer decoder with KV cache |
### Inputs and outputs
* **Input:**
* A CAPTCHA screenshot / image resized and normalized to `768x768` (`CLIPImageProcessor`).
* A Florence-2 task prompt:
* `<OCR>` for distorted-text / OCR CAPTCHAs
* `<CAPTION_TO_PHRASE_GROUNDING><target>` (e.g., `slider puzzle target`, `icon click target`, `text click target`, `drag end target`, `image rotation target`, `visual reasoning target`)
* **Output:**
* Structured action or grounding output, such as `Click(box=(x, y)),Type(content='...')` for `<OCR>` or `<loc_x1><loc_y1><loc_x2><loc_y2>` bounding-box coordinates (0–999 scale) for `<CAPTION_TO_PHRASE_GROUNDING>`.
### Citation
```bibtex
@misc{fiptcha_2026,
title = {Fiptcha: Lightweight CPU-Efficient Local CAPTCHA Solver},
author = {Figranium},
year = {2026},
url = {https://huggingface.co/figranium/fiptcha}
}
```
---
## Model Data
Data used for model training and how the data was processed.
### Training Dataset
Fiptcha was fine-tuned on a combined mixture of three public CAPTCHA datasets (augmented with repetition factors and split into `train.jsonl` and `val.jsonl`), plus an optional local click-target set:
* **[`inclusionAI/VenusBench-CAPTCHA`](https://huggingface.co/datasets/inclusionAI/VenusBench-CAPTCHA)** (`test` split, upsampled `15x`):
* **OCR CAPTCHAs:** Text-based challenges parsed from `action_raw` and annotated with `<OCR>` prompts.
* **Visual / Grounding CAPTCHAs:** Click and interaction target boxes annotated with `<CAPTION_TO_PHRASE_GROUNDING>{prompt}` to predict localized target boxes (`<loc_x1><loc_y1><loc_x2><loc_y2>`).
* **[`OpenCaptchaWorld/Open_CaptchaWorld`](https://huggingface.co/datasets/OpenCaptchaWorld/Open_CaptchaWorld)** (`train` split, upsampled `10x`):
* **Point-Click / Multi-Point CAPTCHAs:** Point coordinates `(x, y)` mapped into padded bounding boxes for visual localization prompts.
* **VQA / Question-Answering CAPTCHAs:** General alphanumeric text and label verification questions mapped to a `<VQA>` prefix and target answers.
* **[`hammer888/captcha-data`](https://huggingface.co/datasets/hammer888/captcha-data)** (`40,000` samples):
* Distorted text-based images representing classic visual OCR CAPTCHAs, mapped to the `<OCR>` task prefix.
### Data Preprocessing
* **Image Preprocessing:** Collected images were cached and normalized into `train.jsonl` / `val.jsonl` catalogs, resized to `768x768` (`resample=3`), and normalized with standard ImageNet/CLIP mean (`[0.485, 0.456, 0.406]`) and std (`[0.229, 0.224, 0.225]`), producing `577` visual tokens.
* **Prompt Formatting:** Each example is formatted as `(image, prefix, suffix)` pairs across `<OCR>`, `<VQA>`, and `<CAPTION_TO_PHRASE_GROUNDING>` task prefixes.
---
## Implementation Information
Details about the model internals and runtime optimization.
* **Architecture:** `Florence2ForConditionalGeneration` — DaViT vision encoder (`depths=[1, 1, 9, 1]`, `dim_embed=[128, 256, 512, 1024]`) paired with a `6`-layer encoder / `6`-layer decoder BART-style language model (`d_model=768`, `12` attention heads, `vocab_size=51,289`).
* **Quantization & Export:** Exported from PyTorch (`attn_implementation="eager"`) to FP32 ONNX (`opset_version=17`) with dynamic batch and sequence axes, then quantized using `onnxruntime.quantization.quantize_dynamic` (`per_channel=True`, `weight_type=QuantType.QInt8`, `MatMulConstBOnly=True`).
---
## Evaluation & Benchmarks
Fiptcha (`ONNX INT8`) is evaluated on the held-out `VenusBench-CAPTCHA` validation split (`val.jsonl`, **44 samples** across 6 visual grounding categories and OCR).
### `VenusBench-CAPTCHA` Validation Results (INT8 ONNX · CPU)
| Category | Samples | IoU@0.5 | Mean IoU |
| :--- | :---: | :---: | :---: |
| **`drag end target`** | 8 | **100.0%** (8/8) | 0.903 |
| **`image rotation target`** | 3 | **100.0%** (3/3) | 0.940 |
| **`text click target`** | 8 | **87.5%** (7/8) | 0.641 |
| **`slider puzzle target`** | 6 | **83.3%** (5/6) | 0.715 |
| **`visual reasoning target`** | 6 | **83.3%** (5/6) | 0.644 |
| **`icon click target`** | 8 | **37.5%** (3/8) | 0.346 |
| **Grounding Overall** | **39** | **79.5%** (31/39) | **0.669** |
- **OCR (`5` samples):** 40.0% (2/5) exact match on `Type(content='...')`.
- **Latency (CPU, greedy `num_beams=1`):** **2.523s** median (**2.526s** mean) per sample using `onnxruntime`.
### Methodology & Setup
- **Task Formulation:** Each sample is evaluated with its corresponding `<CAPTION_TO_PHRASE_GROUNDING><target>` or `<OCR>` prompt using greedy decoding (`num_beams=1`, `max_new_tokens=64`) and `skip_special_tokens=False` to preserve `<loc_...>` coordinate tokens.
- **Grounding Metric:** For each ground-truth bounding box, we take the maximum IoU against all predicted bounding boxes, average across ground-truth boxes in the sample, and report `IoU@0.5` and `Mean IoU`.
---
## Ethics and Safety
Fiptcha is intended for authorized browser automation testing, accessibility workflows, and research on CAPTCHA robustness. Users are responsible for complying with applicable terms of service and laws when automating interactions with web applications.
---
## Usage and Limitations
### Usage
Install the Node.js runtime from [`github.com/figranium/fiptcha`](https://github.com/figranium/fiptcha):
```bash
npm install fiptcha
```
To download the quantized ONNX INT8 model weights locally from Hugging Face:
```bash
hf download figranium/fiptcha --local-dir ./fiptcha-int8
```
### System Requirements
* **Disk Space:** ~431 MB (`vision_encoder.onnx`, `embed_tokens.onnx`, `encoder_model.onnx`, `decoder_model.onnx`, `decoder_with_past_model.onnx`, plus tokenizer/processor files)
* **Hardware:** Standard x86_64 or ARM64 CPU; no GPU required
### Limitations
* **Domain Specificity:** Fine-tuned primarily on UI/web CAPTCHA screenshots (`VenusBench-CAPTCHA`); performance degrades on out-of-domain low-resolution synthetic text CAPTCHAs.
* **Single-Step Inference:** Produces a single `<OCR>` or `<CAPTION_TO_PHRASE_GROUNDING>` prediction per image; multi-round challenge orchestration is handled by the `fiptcha` Node.js runtime.
|