| --- |
| license: mit |
| library_name: onnx |
| datasets: |
| - bxding/recaptcha-390k |
| tags: |
| - computer-vision |
| - image-classification |
| - onnx |
| --- |
| |
| # RC1 |
|
|
| RC1 is a pair of small, fast image recognition models that select the matching |
| cells in reCAPTCHA v2 image grids. The 3×3 model classifies each of the nine |
| cells separately, while the 4×4 model processes the whole grid at once. Both |
| run locally on a CPU with ONNX Runtime. On my $2 VPS, the 3×3 model processes |
| about **9 grids per second** using two CPU threads. |
|
|
|  |
|
|
| *Inference in this demo ran on my $2 VPS. The left window shows a 4×4 grid and |
| the right window shows a 3×3 grid.* |
|
|
| ## Models |
|
|
| | Model | Backbone | Input | Supported targets | |
| | --- | --- | --- | --- | |
| | `grid9.onnx` | MobileNetV4 (`mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k`) | Nine cells from a 3×3 grid | bicycle, bus, car, crosswalk, fire hydrant, motorcycle, parking meter, stairs, traffic light | |
| | `grid16.onnx` | ConvNeXt Tiny (`convnext_tiny.dinov3_lvd1689m`) | One 4×4 grid image | bicycle, bus, chimney, crosswalk, fire hydrant, motorcycle, parking meter, stairs, taxi, tractor, traffic light | |
|
|
| Each model outputs a probability for every cell. The inference script then |
| selects cells using the thresholds stored in the model file. |
|
|
| Both models were fine-tuned on the |
| [recaptcha-390k dataset](https://huggingface.co/datasets/bxding/recaptcha-390k). |
|
|
| ## Running inference |
|
|
| Requires Python 3.11 or newer. From this directory, install the dependencies |
| and run the prediction script: |
|
|
| ```sh |
| python -m pip install onnxruntime numpy pillow |
| python predict_onnx.py --model weights/grid16.onnx --image challenge.jpg --target buses |
| ``` |
|
|
| Use `weights/grid9.onnx` for 3×3 grids and `weights/grid16.onnx` for 4×4 grids. |
|
|
| The script prints JSON with a probability for each cell and a `selected_cells` |
| list. Cells are numbered from 0, left to right and then top to bottom. |
|
|
| To limit the number of CPU threads, pass `--threads`. For all options, run |
| `python predict_onnx.py --help`. |
|
|
| ## Benchmarks |
|
|
| Approximate CPU throughput once the model is loaded. Times include image |
| loading and preprocessing. |
|
|
| | Machine | CPU threads | 3×3 grids/sec | 4×4 grids/sec | |
| | --- | ---: | ---: | ---: | |
| | Apple M2 Pro | 4 | ~10 | ~7 | |
| | $2 VPS | 2 | ~9 | ~4 | |
|
|
| In a separate 3×3 test on the M2 Pro, RC1 ran about **10× faster** than a |
| [ViT fine-tuned on reCAPTCHA](https://huggingface.co/nobodyPerfecZ/vit-finetuned-patch16-224-recaptchav2-v1). |
| Both models were run as float32 ONNX models using four CPU threads. |
|
|
| ## Training and export |
|
|
| The models were trained on an AMD Ryzen Threadripper 9960X with an NVIDIA RTX |
| PRO 6000. |
|
|
| Training requires Python 3.11, [uv](https://docs.astral.sh/uv/), and a local |
| copy of the recaptcha-390k dataset. From this directory: |
|
|
| ```sh |
| # Install dependencies |
| uv sync --locked |
| |
| # Prepare the dataset |
| uv run python train.py prepare --data /path/to/recaptcha-390k --output runs/rc |
| |
| # Train the 3×3 and 4×4 models |
| uv run python train.py train --manifest runs/rc/split.json --output runs/rc --tiles 9 |
| uv run python train.py train --manifest runs/rc/split.json --output runs/rc --tiles 16 --backbone convnext_tiny.dinov3_lvd1689m |
| |
| # Export to ONNX |
| uv run --group export python train.py export --checkpoint runs/rc/grid9/best.pt |
| |
| # Run tests |
| uv run --group export python -m unittest test_training |
| ``` |
|
|
| Notes: |
|
|
| - Training uses a GPU if one is available. On CUDA, add `--amp` to reduce |
| memory usage. |
| - The `prepare` step keeps the dataset's existing splits and leaves out the |
| `excluded` split. |
| - All training outputs are written locally to `runs/`. |
| - For dataset and training options, run `python train.py --help` or add |
| `--help` to any subcommand. |
|
|
| ## Roadmap |
|
|
| - Improve chimney and parking meter recognition with more training data. |
| - Add training data and model support for bridges. |
|
|
| ## Disclaimer |
|
|
| RC1 was created for educational and research purposes. It is not affiliated |
| with or endorsed by Google or reCAPTCHA. Users are responsible for complying |
| with all applicable laws and terms of service. |
|
|
| ## License |
|
|
| RC1 is released under the [MIT License](LICENSE). This license does not cover |
| the source images in the training dataset. |
|
|
