Image-to-Text
Transformers
Safetensors
English
Russian
pp_ocrv6_small_rec
captcha
ocr
minecraft
pp-ocrv6
text-recognition
cyrillic
Instructions to use Titlehhhh/captchaocr-ppocrv6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Titlehhhh/captchaocr-ppocrv6 with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("image-to-text", model="Titlehhhh/captchaocr-ppocrv6")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Titlehhhh/captchaocr-ppocrv6", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Rewrite model card: measured results, held-out table, limitations
Browse files
README.md
CHANGED
|
@@ -18,232 +18,45 @@ base_model:
|
|
| 18 |
- PaddlePaddle/PP-OCRv6_medium_rec_safetensors
|
| 19 |
---
|
| 20 |
|
| 21 |
-
|
| 22 |
|
| 23 |
-
|
| 24 |
-
**extended to 18 762 classes**, so that it reads Cyrillic as well as Latin letters and digits.
|
| 25 |
-
Feed it **one crop of the answer band**, get the string drawn on it.
|
| 26 |
|
| 27 |
-
|
| 28 |
-
> repository.** The card that was here before described an older run (`all20`) whose weights are
|
| 29 |
-
> **no longer** published: 18 710 classes, 15 types, median exact 0.920. Those numbers were true
|
| 30 |
-
> of that model and are **not** true of this one.
|
| 31 |
|
| 32 |
-
|
|
|
|
| 33 |
|
| 34 |
-
|
| 35 |
-
|---|---|
|
| 36 |
-
| `model.safetensors` | the weights: 308 tensors, 19 186 154 float parameters |
|
| 37 |
-
| `config.json` | the base architecture, with `head_out_channels` patched to 18 762 |
|
| 38 |
-
| `preprocessor_config.json` | the base processor + the PaddleOCR normalisation + the 18 762-character dictionary |
|
| 39 |
-
| `processor_config.json` | `PPOCRV6SmallRecImageProcessor` |
|
| 40 |
-
| `character_list.txt` | the dictionary, one entry per line, index 0 = `blank` |
|
| 41 |
-
| `finetune_meta.json` | the provenance of the export (checkpoint, epoch, step, α) |
|
| 42 |
-
| `README.md` | this card |
|
| 43 |
-
|
| 44 |
-
Provenance, as shipped in `finetune_meta.json`:
|
| 45 |
-
|
| 46 |
-
| | |
|
| 47 |
-
|---|---|
|
| 48 |
-
| base model | `PaddlePaddle/PP-OCRv6_medium_rec_safetensors` (Apache-2.0) |
|
| 49 |
-
| fine-tuned checkpoint | `train/runs/ppocrv6/universal_v1/best.pt` |
|
| 50 |
-
| epoch / step | 4 (the last of 5) / 2 440 |
|
| 51 |
-
| classes | **18 762** (index 0 = `blank`) — the pretrained head is **kept and extended**, not replaced |
|
| 52 |
-
| float parameters | **19 186 154** |
|
| 53 |
-
| weight interpolation | **none** — α = 1.0, the checkpoint exactly (see below) |
|
| 54 |
-
| transformers class | `PPOCRV6SmallRecForTextRecognition` |
|
| 55 |
-
| normalisation | mean = std = 0.5 (the PaddleOCR convention) |
|
| 56 |
-
| exported by | `train/export_ppocr_hf.py` |
|
| 57 |
-
|
| 58 |
-
### What changed in the head
|
| 59 |
-
|
| 60 |
-
The pretrained PP-OCRv6 dictionary has **18 710** classes and no Cyrillic letters at all. Rows
|
| 61 |
-
**0 … 18 709** of the published dictionary are the pretrained ones (index 18 709 is the space);
|
| 62 |
-
rows **18 710 … 18 761** are the **52 Cyrillic characters** that were appended:
|
| 63 |
-
|
| 64 |
-
```
|
| 65 |
-
ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ + авдеикмнопрстцчшыья
|
| 66 |
-
```
|
| 67 |
-
|
| 68 |
-
The pretrained rows were **kept**, not replaced (`--expand-charset` grows the head and copies the
|
| 69 |
-
old rows). The extension was done in an earlier run of the same recipe; in this run the flag
|
| 70 |
-
reported «0 characters missing», i.e. the grown head was inherited from the previous checkpoint
|
| 71 |
-
together with its weights.
|
| 72 |
-
|
| 73 |
-
## The training run in numbers
|
| 74 |
-
|
| 75 |
-
| | |
|
| 76 |
-
|---|---|
|
| 77 |
-
| initialised from | `train/runs/ppocrv6/limbo_cyr3/best.pt` — the **«replay mix»** Cyrillic checkpoint (3 short Cyrillic types + digits + 5 Latin types, 4 epochs) |
|
| 78 |
-
| epochs × steps | 5 × 488 = **2 440** steps |
|
| 79 |
-
| batch size | 32 |
|
| 80 |
-
| learning rate | **8e-6** for the backbone **and** 8e-6 for the head |
|
| 81 |
-
| weight decay / grad clip | 0.01 / 1.0 |
|
| 82 |
-
| `--limit-per-type` | **1 500** |
|
| 83 |
-
| training examples | **15 594** |
|
| 84 |
-
| evaluation set | **11 400** images over 18 types |
|
| 85 |
-
| wall clock | **33.9 min** (2 032 s) at **108 img/s** (one GTX 1080 Ti, fp16/AMP) |
|
| 86 |
-
| seed | 42 · `--workers 0` (the sandbox this was trained in denies multiprocessing named pipes) |
|
| 87 |
-
|
| 88 |
-
### What was trained on, and what was not
|
| 89 |
-
|
| 90 |
-
**12 types were trained on.** Eight of them are fed as the **answer band** (the `roi` rectangle
|
| 91 |
-
of the dataset); the four that have no band at all are trained on the **whole frame**, so the
|
| 92 |
-
model has to find the code by itself:
|
| 93 |
-
|
| 94 |
-
| type | input |
|
| 95 |
-
|---|---|
|
| 96 |
-
| `out/Archy-X/Krypton/map` | band |
|
| 97 |
-
| `out/haq/mapcha/map` | band |
|
| 98 |
-
| `out/just-captcha/ProstoCaptcha/map` | band |
|
| 99 |
-
| `out/ringoame196-s-mcPlugin/ReproductionCAPTCHA/map` | band |
|
| 100 |
-
| `out_ru/GalleryCaptcha/ru-symbols` | band |
|
| 101 |
-
| `out_ru/GalleryCaptcha/ru-web` | band |
|
| 102 |
-
| `out_ru/LimboFilter/digits` | band |
|
| 103 |
-
| `out_ru/LimboFilter/symbols` | band |
|
| 104 |
-
| `out/InstantlyMoist/Captcha/textcaptcha` | **whole frame** |
|
| 105 |
-
| `out/LagBug/XProtect/code` | **whole frame** |
|
| 106 |
-
| `out/LagBug/XProtect/map` | **whole frame** |
|
| 107 |
-
| `out/irootdev/AnBot/captcha` | **whole frame** |
|
| 108 |
-
|
| 109 |
-
(The names above are the dataset's own `type_id` values.)
|
| 110 |
-
|
| 111 |
-
**6 types were never seen in training.** The unit of generalisation is the **plugin**, never the
|
| 112 |
-
picture: these six whole plugins were held out of the training set:
|
| 113 |
-
|
| 114 |
-
`out/JSH32/Minecaptcha/map`, `out/jonesdevelopment/sonar/standard`,
|
| 115 |
-
`out/jonesdevelopment/sonar/legacy`, `out/jonesdevelopment/sonar/complex`,
|
| 116 |
-
`out_ru/BotFilter/map`, `out_ru/CaptchaVerifyZ/map`.
|
| 117 |
-
|
| 118 |
-
**3 types were excluded on purpose:** `out_ru/GalleryCaptcha/ru-numberword`,
|
| 119 |
-
`out_ru/GalleryCaptcha/ru-numberword-caption` and `out_ru/LimboFilter/map`. In all three the picture shows a
|
| 120 |
-
Russian number **written out in words** while the player types **digits** (`text != answer` in
|
| 121 |
-
all 10 000 rows of each). Converting a written number into a typed one is a **solver step**,
|
| 122 |
-
not an OCR task, so the model was not asked to do it. That is a limitation of the *task*, not a
|
| 123 |
-
defect of the model — but it does mean these three types are **not supported**.
|
| 124 |
-
|
| 125 |
-
## Measured results
|
| 126 |
-
|
| 127 |
-
Measured inside the same run (`train/runs/ppocrv6/universal_v1/result.json`): 11 400 images over
|
| 128 |
-
18 types, greedy CTC decoding, exact match against the drawn string.
|
| 129 |
-
|
| 130 |
-
| aggregate | before this run¹ | **after** |
|
| 131 |
-
|---|---|---|
|
| 132 |
-
| median exact match over 18 types | 0.8750 | **0.8875** |
|
| 133 |
-
| median CER | 0.0394 | **0.0344** |
|
| 134 |
-
| micro exact match (11 400 images) | 0.6499 | **0.7797** |
|
| 135 |
-
| micro CER | 0.1795 | **0.0755** |
|
| 136 |
-
|
| 137 |
-
¹ «before this run» is the **zero step of this very run** — the checkpoint it was initialised
|
| 138 |
-
from (`limbo_cyr3/best.pt`), **not** the pretrained PP-OCRv6 base. The fine-tune moved the
|
| 139 |
-
median only a little (0.875 → 0.8875) but the **micro** average a lot (0.6499 → 0.7797): the
|
| 140 |
-
median is dominated by the types that were already solved, while the micro average counts every
|
| 141 |
-
image — and the frame-trained types that were unreadable rose sharply
|
| 142 |
-
(`out/LagBug/XProtect/code` 0.00 → 0.36, `out/LagBug/XProtect/map` 0.00 → 0.27,
|
| 143 |
-
`out/InstantlyMoist/Captcha/textcaptcha` 0.12 → 0.24).
|
| 144 |
-
|
| 145 |
-
### Types the model was trained on
|
| 146 |
-
|
| 147 |
-
These numbers are a **sample-level hold-out**: the first 200 rows of every type were taken out
|
| 148 |
-
of training, the rest were trained on. They are *not* unseen-plugin numbers.
|
| 149 |
-
|
| 150 |
-
| type | n | exact (zero step) | **exact** | CER |
|
| 151 |
-
|---|---:|---:|---:|---:|
|
| 152 |
-
| `out/Archy-X/Krypton/map` | 200 | 1.0000 | **1.0000** | 0.0000 |
|
| 153 |
-
| `out/ringoame196-s-mcPlugin/ReproductionCAPTCHA/map` | 200 | 1.0000 | **1.0000** | 0.0000 |
|
| 154 |
-
| `out/irootdev/AnBot/captcha` | 200 | 0.9350 | **0.9950** | 0.0017 |
|
| 155 |
-
| `out/haq/mapcha/map` | 200 | 0.9650 | 0.9550 | 0.0112 |
|
| 156 |
-
| `out_ru/GalleryCaptcha/ru-web` | 200 | 0.9300 | 0.9250 | 0.0150 |
|
| 157 |
-
| `out/just-captcha/ProstoCaptcha/map` | 200 | 0.9250 | 0.9150 | 0.0150 |
|
| 158 |
-
| `out_ru/LimboFilter/digits` | 200 | 0.8900 | 0.8900 | 0.0288 |
|
| 159 |
-
| `out_ru/GalleryCaptcha/ru-symbols` | 200 | 0.8600 | **0.8850** | 0.0400 |
|
| 160 |
-
| `out_ru/LimboFilter/symbols` | 200 | 0.7650 | **0.8000** | 0.0783 |
|
| 161 |
-
| `out/LagBug/XProtect/code` | 200 | 0.0000 | **0.3600** | 0.1542 |
|
| 162 |
-
| `out/LagBug/XProtect/map` | 200 | 0.0000 | **0.2700** | 0.1850 |
|
| 163 |
-
| `out/InstantlyMoist/Captcha/textcaptcha` | 200 | 0.1200 | **0.2400** | 0.4220 |
|
| 164 |
-
|
| 165 |
-
### Types the model never saw
|
| 166 |
-
|
| 167 |
-
Whole plugins that were held out of training — these are the honest generalisation numbers.
|
| 168 |
-
|
| 169 |
-
| type | n | exact (zero step) | **exact** | CER |
|
| 170 |
-
|---|---:|---:|---:|---:|
|
| 171 |
-
| `out_ru/CaptchaVerifyZ/map` | 1 500 | 0.9647 | **0.9780** | 0.0045 |
|
| 172 |
-
| `out/JSH32/Minecaptcha/map` | 1 500 | 0.9340 | **0.9373** | 0.0162 |
|
| 173 |
-
| `out_ru/BotFilter/map` | 1 500 | 0.7113 | **0.8140** | 0.0508 |
|
| 174 |
-
| `out/jonesdevelopment/sonar/legacy` | 1 500 | 0.5973 | **0.7693** | 0.0793 |
|
| 175 |
-
| `out/jonesdevelopment/sonar/complex` | 1 500 | 0.3520 | **0.6253** | 0.1293 |
|
| 176 |
-
| `out/jonesdevelopment/sonar/standard` | 1 500 | 0.2613 | **0.5707** | 0.1671 |
|
| 177 |
-
|
| 178 |
-
**One caveat about the protocol.** The evaluation runs on held-out rows of the **same generator
|
| 179 |
-
render** the model was trained on (the generators are not seeded, so a re-render is a different
|
| 180 |
-
sample set). For the twelve trained types the hold-out is by *sample*; only the six held-out
|
| 181 |
-
plugins above are genuinely unseen styles.
|
| 182 |
-
|
| 183 |
-
## No weight interpolation — and why the previous card said α = 0.4
|
| 184 |
-
|
| 185 |
-
The previous version of this repository published a WiSE-FT soup, `θ = 0.4·θ_ft + 0.6·θ_pretrained`.
|
| 186 |
-
That was the right call **for that model**: a small fine-tune on a few types degraded the unseen
|
| 187 |
-
plugins, and interpolating back towards the base restored them (measured there: base median
|
| 188 |
-
0.588 → α = 0.4 median 0.638 → α = 1.0 median 0.480).
|
| 189 |
-
|
| 190 |
-
This model is different in two ways: it is trained on **most** of the types it is evaluated on,
|
| 191 |
-
and its head has **grown**. On an expanded head the interpolation has nothing to interpolate
|
| 192 |
-
*towards* for the 52 appended rows — the base model has no such rows — while the encoder is
|
| 193 |
-
dragged back to the pretrained solution. Measured on the α-sweep of the `limbo_cyr3`
|
| 194 |
-
predecessor: the sweep's best α = 0.6 lifted the Latin held-out types from 0.644 to 0.772 and
|
| 195 |
-
at the same time **collapsed the Cyrillic ones**:
|
| 196 |
-
|
| 197 |
-
| type | α = 1.0 | after interpolation (α = 0.6) |
|
| 198 |
-
|---|---:|---:|
|
| 199 |
-
| `out_ru/LimboFilter/symbols` | 0.765 | **0.016** |
|
| 200 |
-
| `out_ru/GalleryCaptcha/ru-symbols` | 0.860 | **0.002** |
|
| 201 |
-
| `out_ru/LimboFilter/digits` | 0.890 | **0.666** |
|
| 202 |
-
|
| 203 |
-
So WiSE-FT is **destructive** on an expanded head. That soup was kept as an exhibit
|
| 204 |
-
(`train/runs/ppocrv6/limbo_cyr3/soup_best.pt`) and is **not** used here. What works for a
|
| 205 |
-
vocabulary expansion on this model is **replay** — keeping the old data in the training mix —
|
| 206 |
-
and this checkpoint is therefore published with **α = 1.0**: the fine-tuned weights, unchanged.
|
| 207 |
-
|
| 208 |
-
## What this model is NOT good at
|
| 209 |
-
|
| 210 |
-
* **It is not a universal captcha solver.** It is a fine-tune for a **known set of captcha
|
| 211 |
-
styles**: a plugin that draws in a style absent from the training set will not be handled.
|
| 212 |
-
For the six plugins that were held out, the honest expectation is the numbers in the table
|
| 213 |
-
above — between 0.57 and 0.98 exact match, not 1.0.
|
| 214 |
-
* **The three number-word types are not supported at all** (`out_ru/GalleryCaptcha/ru-numberword`,
|
| 215 |
-
`out_ru/GalleryCaptcha/ru-numberword-caption`, `out_ru/LimboFilter/map`): the picture shows a number
|
| 216 |
-
written in Russian words while the answer is digits. Use a solver step for those.
|
| 217 |
-
* **The weakest trained types are `out/LagBug/XProtect/code` (0.36) and `out/LagBug/XProtect/map`
|
| 218 |
-
(0.27)**: both are the same 130×130 sheet with per-character cumulative translation and 51
|
| 219 |
-
interference lines. `out/InstantlyMoist/Captcha/textcaptcha` (0.24) is a 128×128 sheet with
|
| 220 |
-
thin diagonal noise.
|
| 221 |
-
* **It reads a band, not a picture.** On a whole game screenshot it falls apart — see below.
|
| 222 |
-
|
| 223 |
-
## How to use it
|
| 224 |
|
| 225 |
```python
|
| 226 |
import torch
|
| 227 |
from PIL import Image
|
| 228 |
-
from transformers import PPOCRV6SmallRecForTextRecognition
|
|
|
|
| 229 |
|
| 230 |
REPO = "Titlehhhh/captchaocr-ppocrv6"
|
| 231 |
-
|
| 232 |
-
model = PPOCRV6SmallRecForTextRecognition.from_pretrained(REPO)
|
| 233 |
proc = PPOCRV6SmallRecImageProcessor.from_pretrained(REPO)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 234 |
|
| 235 |
-
|
| 236 |
-
|
|
|
|
| 237 |
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
|
|
|
| 241 |
|
| 242 |
-
|
| 243 |
with open("character_list.txt", encoding="utf-8") as f:
|
| 244 |
dictionary = [line.rstrip("\r\n") for line in f] # rstrip, not strip: " " is a class
|
| 245 |
-
|
| 246 |
-
ids = logits.argmax(-1)[0].tolist()
|
| 247 |
text, prev = [], -1
|
| 248 |
for i in ids:
|
| 249 |
if i != prev and i != 0: # collapse repeats, drop the blank
|
|
@@ -251,94 +64,116 @@ for i in ids:
|
|
| 251 |
prev = i
|
| 252 |
print("".join(text))
|
| 253 |
```
|
|
|
|
| 254 |
|
| 255 |
-
|
| 256 |
-
can decode without the text file. The input geometry is 48 × 320 (resize + pad), and the
|
| 257 |
-
normalisation is `mean = std = 0.5` — the PaddleOCR convention the fine-tune was trained with.
|
| 258 |
-
**Do not let the processor fall back to the ImageNet statistics**: `preprocessor_config.json`
|
| 259 |
-
already carries the patched values.
|
| 260 |
|
| 261 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 262 |
|
| 263 |
-
|
| 264 |
-
code. On a **whole game screenshot** it collapses: measured on the six real screenshots of the
|
| 265 |
-
project owner, feeding the whole frame gives **2/6** correct readings.
|
| 266 |
|
| 267 |
-
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
cropper: **6/6** on those same six screenshots. The implementation ships with the project
|
| 271 |
-
(`webapp/cropper.py`, `webapp/app.py` — see
|
| 272 |
-
<https://github.com/Titlehhhh/CaptchaOCR> and `reports/CROPPER.md` for the measured numbers).
|
| 273 |
|
| 274 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 275 |
|
| 276 |
-
|
| 277 |
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
|
| 281 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 282 |
|
| 283 |
-
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
|
| 287 |
-
| files | 9 Parquet shards (`data/train-*-of-00009.parquet`), one row per sample, raw PNG in `image` |
|
| 288 |
-
| bands | 12 types carry an `roi` band; 119 984 rows have one, 80 016 do not |
|
| 289 |
-
| `text != answer` | 3 types (the Russian number words) |
|
| 290 |
-
| alphabet | 115 distinct characters over the whole set |
|
| 291 |
-
|
| 292 |
-
Two facts worth stating plainly:
|
| 293 |
-
|
| 294 |
-
* The model was trained on the **project's own render** of that dataset (10 000 images per
|
| 295 |
-
type), not on the published Parquet shards — the pixels are the same style, the sample set is
|
| 296 |
-
that of the local render.
|
| 297 |
-
* The type **`out_ru/LimboFilter/digits`** (four digits, real-server style) was trained on and
|
| 298 |
-
**is not part of the published dataset snapshot**: the generator gained that type after the
|
| 299 |
-
dataset was exported.
|
| 300 |
-
|
| 301 |
-
### Provenance and licence
|
| 302 |
-
|
| 303 |
-
* Base model: `PaddlePaddle/PP-OCRv6_medium_rec_safetensors`, **Apache-2.0**. These weights are
|
| 304 |
-
a derivative of it.
|
| 305 |
-
* The dataset is **synthetic** — re-renders, not screenshots of a live server; no picture, page
|
| 306 |
-
or answer of a third-party commercial service is used or published.
|
| 307 |
-
* The dataset is published under **the licence of the upstream plugin each row came from**
|
| 308 |
-
(`license_name: mixed-per-plugin-licences`; the per-plugin breakdown is in the dataset
|
| 309 |
-
repository's `LICENSE` and in its card). Some of those licences forbid commercial hosting or
|
| 310 |
-
reserve all rights — **read them before you use the data**.
|
| 311 |
-
* The 52 characters added to the head are the Cyrillic alphabet; the images that taught them
|
| 312 |
-
come from the `GalleryCaptcha` and `LimboFilter` types, whose licences are listed above.
|
| 313 |
-
|
| 314 |
-
## Reproducing
|
| 315 |
-
|
| 316 |
-
```powershell
|
| 317 |
-
$env:PYTHONIOENCODING="utf-8"
|
| 318 |
-
$env:HF_HOME="C:\Users\Title\Projects\CapthaModel\.hf-cache"
|
| 319 |
-
|
| 320 |
-
# the run that produced these weights
|
| 321 |
-
.venv-gpu\Scripts\python.exe train\finetune_ppocr.py --out train\runs\ppocrv6\universal_v1 `
|
| 322 |
-
--init-from train\runs\ppocrv6\limbo_cyr3\best.pt --expand-charset --train-frames `
|
| 323 |
-
--epochs 5 --batch-size 32 --lr 8e-6 --head-lr 8e-6 --workers 0 --seed 42 `
|
| 324 |
-
--limit-per-type 1500 --holdout-per-type 200 --eval-every 5 --save-every 400 --max-minutes 60 `
|
| 325 |
-
--val-types out/jonesdevelopment/sonar/standard out/jonesdevelopment/sonar/legacy out/jonesdevelopment/sonar/complex out_ru/BotFilter/map out_ru/CaptchaVerifyZ/map out/laforetbrut/are-you-sure/code out/JSH32/Minecaptcha/map `
|
| 326 |
-
--exclude-types ru-numberword ru-numberword-caption LimboFilter/map
|
| 327 |
-
|
| 328 |
-
# the export (alpha = 1.0 -> the checkpoint itself, no interpolation)
|
| 329 |
-
.venv-gpu\Scripts\python.exe train\export_ppocr_hf.py `
|
| 330 |
-
--ckpt train\runs\ppocrv6\universal_v1\best.pt --soup-alpha 1.0 `
|
| 331 |
-
--out train\runs\ppocrv6\universal_v1\export_hf
|
| 332 |
-
```
|
| 333 |
|
| 334 |
-
|
| 335 |
-
|
| 336 |
-
|
| 337 |
-
|
| 338 |
-
|
| 339 |
-
|
| 340 |
-
|
| 341 |
-
|
| 342 |
-
|
| 343 |
-
|
| 344 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
- PaddlePaddle/PP-OCRv6_medium_rec_safetensors
|
| 19 |
---
|
| 20 |
|
| 21 |
+
<img src="https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset/resolve/main/assets/model_hero.png" alt="A haq/mapcha/map captcha frame, the answer band cropped from it by roi, and the string the model decodes from that band: ly8r" width="100%">
|
| 22 |
|
| 23 |
+
       
|
|
|
|
|
|
|
| 24 |
|
| 25 |
+
# CaptchaOCR PP-OCRv6 — captcha text recogniser
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
+
A PP-OCRv6 fine-tune that reads a captcha image, usually the answer band: 18 evaluated styles,
|
| 28 |
+
Latin letters, digits, and part of the Cyrillic alphabet.
|
| 29 |
|
| 30 |
+
## Use it in 10 lines
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
```python
|
| 33 |
import torch
|
| 34 |
from PIL import Image
|
| 35 |
+
from transformers import PPOCRV6SmallRecForTextRecognition
|
| 36 |
+
from transformers import PPOCRV6SmallRecImageProcessor
|
| 37 |
|
| 38 |
REPO = "Titlehhhh/captchaocr-ppocrv6"
|
| 39 |
+
model = PPOCRV6SmallRecForTextRecognition.from_pretrained(REPO).eval()
|
|
|
|
| 40 |
proc = PPOCRV6SmallRecImageProcessor.from_pretrained(REPO)
|
| 41 |
+
inputs = proc(images=[Image.open("band.png").convert("RGB")], return_tensors="pt")
|
| 42 |
+
with torch.no_grad():
|
| 43 |
+
logits = model(**inputs)
|
| 44 |
+
print(proc.post_process_text_recognition(logits)) # [{'text': 'ly8r', 'score': 0.96}]
|
| 45 |
+
```
|
| 46 |
|
| 47 |
+
`band.png` is a tight crop of the answer band, not a screenshot. The processor resizes it to
|
| 48 |
+
48 × 320, normalises it, and decodes CTC with the 18 762-entry dictionary from
|
| 49 |
+
`preprocessor_config.json`. This exact snippet was run against these weights.
|
| 50 |
|
| 51 |
+
<details>
|
| 52 |
+
<summary>Manual CTC decoding</summary>
|
| 53 |
+
|
| 54 |
+
If you would rather decode by hand, the forward returns the logits in `last_hidden_state`:
|
| 55 |
|
| 56 |
+
```python
|
| 57 |
with open("character_list.txt", encoding="utf-8") as f:
|
| 58 |
dictionary = [line.rstrip("\r\n") for line in f] # rstrip, not strip: " " is a class
|
| 59 |
+
ids = logits.last_hidden_state.argmax(-1)[0].tolist()
|
|
|
|
| 60 |
text, prev = [], -1
|
| 61 |
for i in ids:
|
| 62 |
if i != prev and i != 0: # collapse repeats, drop the blank
|
|
|
|
| 64 |
prev = i
|
| 65 |
print("".join(text))
|
| 66 |
```
|
| 67 |
+
</details>
|
| 68 |
|
| 69 |
+
## What it supports
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
|
| 71 |
+
| Supports | Does not support |
|
| 72 |
+
|---|---|
|
| 73 |
+
| **18 evaluated styles**: 11 from the published dataset snapshot, 1 unpublished (`LimboFilter/digits`), 6 held-out plugins | **3 types that draw a number as a word** while the answer is digits: `ru-numberword`, `ru-numberword-caption`, `LimboFilter/map`. Those need a solver step, not OCR |
|
| 74 |
+
| Latin letters, digits, and 52 of 66 Cyrillic letters: all 33 uppercase, 19 lowercase | 14 lowercase Cyrillic letters are not classes: б, г, ё, ж, з, й, л, у, ф, х, щ, ъ, э, ю. Characters outside `character_list.txt` are never produced |
|
| 75 |
+
| A tight crop of the answer band (`roi` in the dataset), noisy bands with lines, per-character shifts, coloured backgrounds | A whole game screenshot without a cropper: 2 of 6 screenshots read correctly as-is, 6 of 6 with the project's cropper |
|
| 76 |
+
| Styles close to the trained plugins, including other plugins of the same visual family | A style absent from training: expect 0.57–0.98 exact, never 1.0 |
|
| 77 |
|
| 78 |
+
## Results
|
|
|
|
|
|
|
| 79 |
|
| 80 |
+
Exact match and CER after greedy CTC decoding, on 11 400 images of 18 styles: 200 rows per trained
|
| 81 |
+
style plus 1 500 rows per held-out plugin. The median is taken over the 18 per-style scores; the
|
| 82 |
+
micro average counts every image, and 9 000 of the 11 400 belong to held-out plugins.
|
|
|
|
|
|
|
|
|
|
| 83 |
|
| 84 |
+
| Metric | Score |
|
| 85 |
+
|---|---:|
|
| 86 |
+
| median exact match (18 styles) | **0.8875** |
|
| 87 |
+
| median CER | **0.0344** |
|
| 88 |
+
| micro exact match (11 400 images) | **0.7797** |
|
| 89 |
+
| micro CER | **0.0755** |
|
| 90 |
|
| 91 |
+
**Plugins never seen in training** — what to expect on a new style:
|
| 92 |
|
| 93 |
+
| Plugin | Style | Exact | CER |
|
| 94 |
+
|---|---|---:|---:|
|
| 95 |
+
| CaptchaVerifyZ | map | 0.9780 | 0.0045 |
|
| 96 |
+
| Minecaptcha | map | 0.9373 | 0.0162 |
|
| 97 |
+
| BotFilter | map | 0.8140 | 0.0508 |
|
| 98 |
+
| sonar | legacy | 0.7693 | 0.0793 |
|
| 99 |
+
| sonar | complex | 0.6253 | 0.1293 |
|
| 100 |
+
| sonar | standard | 0.5707 | 0.1671 |
|
| 101 |
|
| 102 |
+
<details>
|
| 103 |
+
<summary>Per-style results on the trained styles</summary>
|
| 104 |
+
|
| 105 |
+
Sample-level hold-out: the first 200 rows of each style were taken out of training.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|
| 107 |
+
| Style | Exact | CER |
|
| 108 |
+
|---|---:|---:|
|
| 109 |
+
| `Archy-X/Krypton/map` | 1.0000 | 0.0000 |
|
| 110 |
+
| `ringoame196-s-mcPlugin/ReproductionCAPTCHA/map` | 1.0000 | 0.0000 |
|
| 111 |
+
| `irootdev/AnBot/captcha` | 0.9950 | 0.0017 |
|
| 112 |
+
| `haq/mapcha/map` | 0.9550 | 0.0112 |
|
| 113 |
+
| `GalleryCaptcha/ru-web` | 0.9250 | 0.0150 |
|
| 114 |
+
| `just-captcha/ProstoCaptcha/map` | 0.9150 | 0.0150 |
|
| 115 |
+
| `LimboFilter/digits` (unpublished) | 0.8900 | 0.0288 |
|
| 116 |
+
| `GalleryCaptcha/ru-symbols` | 0.8850 | 0.0400 |
|
| 117 |
+
| `LimboFilter/symbols` | 0.8000 | 0.0783 |
|
| 118 |
+
| `LagBug/XProtect/code` | 0.3600 | 0.1542 |
|
| 119 |
+
| `LagBug/XProtect/map` | 0.2700 | 0.1850 |
|
| 120 |
+
| `InstantlyMoist/Captcha/textcaptcha` | 0.2400 | 0.4220 |
|
| 121 |
+
|
| 122 |
+
</details>
|
| 123 |
+
|
| 124 |
+
## Limitations
|
| 125 |
+
|
| 126 |
+
* **Four of the twelve trained styles are fed the whole frame**, not a band. Three of them are the
|
| 127 |
+
weakest: `XProtect/code` 0.36, `XProtect/map` 0.27, `textcaptcha` 0.24. The fourth,
|
| 128 |
+
`AnBot/captcha`, scores 0.9950 on the frame — the input is not what makes a style hard.
|
| 129 |
+
* **The median is carried by the easy styles.** Seven of the twelve trained styles score 0.89 or
|
| 130 |
+
better. On a style unlike the training set, read the held-out table instead: 0.57–0.98.
|
| 131 |
+
* **Screenshots need a cropper.** Feed a raw game frame and 2 of 6 real screenshots are read
|
| 132 |
+
correctly; with the cropper, 6 of 6. The cropper ships with the code, not with these weights.
|
| 133 |
+
* **The micro average is mostly held-out data** (9 000 of 11 400 images), so it answers "how does
|
| 134 |
+
it do on unseen plugins", not "accuracy in general".
|
| 135 |
+
* **It is not an API and not a solver for new plugins.** One image in, one string out; a style
|
| 136 |
+
absent from training has no guarantee.
|
| 137 |
+
|
| 138 |
+
## Dataset
|
| 139 |
+
|
| 140 |
+
[`Titlehhhh/captchaocr-dataset`](https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset) is
|
| 141 |
+
synthetic: every image is a re-render by this project's Java generators of a captcha drawn by an
|
| 142 |
+
open-source Minecraft plugin. The published snapshot holds 20 styles and 200 000 samples. A 21st
|
| 143 |
+
style, `out_ru/LimboFilter/digits`, was generated after the export and is not in the snapshot —
|
| 144 |
+
which is why 18 evaluated styles = 11 from the snapshot + 1 unpublished + 6 held out.
|
| 145 |
+
|
| 146 |
+
<details>
|
| 147 |
+
<summary>Training</summary>
|
| 148 |
+
|
| 149 |
+
* Base: `PP-OCRv6_medium_rec` (Apache-2.0). The CTC head was kept and grown from 18 710 to
|
| 150 |
+
18 762 classes: rows 18 710–18 761 are 52 appended Cyrillic characters.
|
| 151 |
+
* Initialised from an earlier Cyrillic run of the same recipe that replayed Latin data, then
|
| 152 |
+
fine-tuned on 12 styles.
|
| 153 |
+
* 5 epochs × 488 steps = 2 440 steps, batch 32, learning rate 8e-6 for backbone and head,
|
| 154 |
+
weight decay 0.01, gradient clipping 1.0.
|
| 155 |
+
* 15 594 training examples: 1 300 per style after a 200-row hold-out, from a 10 000-per-style
|
| 156 |
+
render (the run used `--limit-per-type 1500`).
|
| 157 |
+
* 33.9 min on one GTX 1080 Ti, 108 images/s, fp16.
|
| 158 |
+
* Whole-frame styles were trained with the frame as input; the rest with the `roi` band.
|
| 159 |
+
* Full command line and scripts:
|
| 160 |
+
[github.com/Titlehhhh/CaptchaOCR](https://github.com/Titlehhhh/CaptchaOCR).
|
| 161 |
+
|
| 162 |
+
</details>
|
| 163 |
+
|
| 164 |
+
## License & citation
|
| 165 |
+
|
| 166 |
+
The weights are **Apache-2.0**, as a derivative of `PP-OCRv6_medium_rec`. The training data is
|
| 167 |
+
published under **mixed per-plugin licences** — some rows are non-commercial, so read the dataset
|
| 168 |
+
[LICENSE](https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset/blob/main/LICENSE) before
|
| 169 |
+
commercial use.
|
| 170 |
+
|
| 171 |
+
<details>
|
| 172 |
+
<summary>Version history</summary>
|
| 173 |
+
|
| 174 |
+
* `universal_v1` (2026-10-27) — current weights, 18 762 classes, no weight interpolation (α = 1.0).
|
| 175 |
+
It replaced `all20`, which is no longer downloadable.
|
| 176 |
+
* Interpolation is off on purpose: pulling back towards the base damages the appended rows
|
| 177 |
+
(`LimboFilter/symbols` 0.765 → 0.016). Replay of old data, not a weight soup, is what keeps them.
|
| 178 |
+
|
| 179 |
+
</details>
|