Image-to-Text
Transformers
Safetensors
English
Russian
pp_ocrv6_small_rec
captcha
ocr
minecraft
pp-ocrv6
text-recognition
cyrillic
Instructions to use Titlehhhh/captchaocr-ppocrv6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Titlehhhh/captchaocr-ppocrv6 with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("image-to-text", model="Titlehhhh/captchaocr-ppocrv6")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Titlehhhh/captchaocr-ppocrv6", device_map="auto") - Notebooks
- Google Colab
- Kaggle
model card: describe universal_v1, not the previous all20 run
Browse files
README.md
CHANGED
|
@@ -11,6 +11,7 @@ tags:
|
|
| 11 |
- minecraft
|
| 12 |
- pp-ocrv6
|
| 13 |
- text-recognition
|
|
|
|
| 14 |
datasets:
|
| 15 |
- Titlehhhh/captchaocr-dataset
|
| 16 |
base_model:
|
|
@@ -19,96 +20,205 @@ base_model:
|
|
| 19 |
|
| 20 |
# CaptchaOCR PP-OCRv6 — a text recogniser for Minecraft anti-bot captchas
|
| 21 |
|
| 22 |
-
A fine-tune of **PP-OCRv6_medium_rec** on
|
| 23 |
-
|
| 24 |
-
Feed it one
|
| 25 |
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
-
|
| 29 |
-
Every plugin draws differently: its own font, canvas size, noise, distortion — and its own bugs.
|
| 30 |
-
The 20 types in the dataset are flat captcha images; this model was trained on the **15** of them
|
| 31 |
-
whose alphabet fits the PP-OCRv6 dictionary.
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
`
|
| 37 |
-
|
| 38 |
-
`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
-
## What
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
| | |
|
| 47 |
|---|---|
|
| 48 |
-
|
|
| 49 |
-
|
|
| 50 |
-
|
|
| 51 |
-
|
|
| 52 |
-
|
|
| 53 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
-
##
|
| 56 |
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
otherwise; greedy CTC decode; exact match on `text` (letters/digits of any script plus `+-*/=`).
|
| 61 |
|
| 62 |
-
|
|
| 63 |
|---|---|
|
| 64 |
-
|
|
| 65 |
-
|
|
| 66 |
-
|
|
| 67 |
-
|
|
| 68 |
-
|
| 69 |
-
|
|
| 70 |
-
|
|
| 71 |
-
| `
|
| 72 |
-
| `
|
| 73 |
-
| `
|
| 74 |
-
| `
|
| 75 |
-
| `
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
| α | median exact | CER |
|
| 100 |
|---|---|---|
|
| 101 |
-
|
|
| 102 |
-
|
|
| 103 |
-
|
|
| 104 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
-
|
|
|
|
|
|
|
|
|
|
| 107 |
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
## How to use it
|
| 114 |
|
|
@@ -117,34 +227,118 @@ import torch
|
|
| 117 |
from PIL import Image
|
| 118 |
from transformers import PPOCRV6SmallRecForTextRecognition, PPOCRV6SmallRecImageProcessor
|
| 119 |
|
| 120 |
-
|
| 121 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
|
| 123 |
-
image = Image.open("001.png").convert("RGB")
|
| 124 |
-
px = proc(images=[image], return_tensors="pt")["pixel_values"]
|
| 125 |
with torch.no_grad():
|
| 126 |
logits = model.head.head(model.head.encoder(
|
| 127 |
model.model(px).last_hidden_state).last_hidden_state)
|
| 128 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
```
|
| 130 |
|
| 131 |
-
The
|
| 132 |
-
|
| 133 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
|
| 135 |
-
|
| 136 |
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 141 |
|
| 142 |
## Reproducing
|
| 143 |
|
| 144 |
```powershell
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 150 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
- minecraft
|
| 12 |
- pp-ocrv6
|
| 13 |
- text-recognition
|
| 14 |
+
- cyrillic
|
| 15 |
datasets:
|
| 16 |
- Titlehhhh/captchaocr-dataset
|
| 17 |
base_model:
|
|
|
|
| 20 |
|
| 21 |
# CaptchaOCR PP-OCRv6 — a text recogniser for Minecraft anti-bot captchas
|
| 22 |
|
| 23 |
+
A fine-tune of **PP-OCRv6_medium_rec** on Minecraft anti-bot captchas whose CTC head was
|
| 24 |
+
**extended to 18 762 classes**, so that it reads Cyrillic as well as Latin letters and digits.
|
| 25 |
+
Feed it **one crop of the answer band**, get the string drawn on it.
|
| 26 |
|
| 27 |
+
> **This card describes the `universal_v1` checkpoint — the weights that are actually in this
|
| 28 |
+
> repository.** The card that was here before described an older run (`all20`) whose weights are
|
| 29 |
+
> **no longer** published: 18 710 classes, 15 types, median exact 0.920. Those numbers were true
|
| 30 |
+
> of that model and are **not** true of this one.
|
| 31 |
|
| 32 |
+
## What is in this repository
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
+
| file | what it is |
|
| 35 |
+
|---|---|
|
| 36 |
+
| `model.safetensors` | the weights: 308 tensors, 19 186 154 float parameters |
|
| 37 |
+
| `config.json` | the base architecture, with `head_out_channels` patched to 18 762 |
|
| 38 |
+
| `preprocessor_config.json` | the base processor + the PaddleOCR normalisation + the 18 762-character dictionary |
|
| 39 |
+
| `processor_config.json` | `PPOCRV6SmallRecImageProcessor` |
|
| 40 |
+
| `character_list.txt` | the dictionary, one entry per line, index 0 = `blank` |
|
| 41 |
+
| `finetune_meta.json` | the provenance of the export (checkpoint, epoch, step, α) |
|
| 42 |
+
| `README.md` | this card |
|
| 43 |
+
|
| 44 |
+
Provenance, as shipped in `finetune_meta.json`:
|
| 45 |
|
| 46 |
+
| | |
|
| 47 |
+
|---|---|
|
| 48 |
+
| base model | `PaddlePaddle/PP-OCRv6_medium_rec_safetensors` (Apache-2.0) |
|
| 49 |
+
| fine-tuned checkpoint | `train/runs/ppocrv6/universal_v1/best.pt` |
|
| 50 |
+
| epoch / step | 4 (the last of 5) / 2 440 |
|
| 51 |
+
| classes | **18 762** (index 0 = `blank`) — the pretrained head is **kept and extended**, not replaced |
|
| 52 |
+
| float parameters | **19 186 154** |
|
| 53 |
+
| weight interpolation | **none** — α = 1.0, the checkpoint exactly (see below) |
|
| 54 |
+
| transformers class | `PPOCRV6SmallRecForTextRecognition` |
|
| 55 |
+
| normalisation | mean = std = 0.5 (the PaddleOCR convention) |
|
| 56 |
+
| exported by | `train/export_ppocr_hf.py` |
|
| 57 |
|
| 58 |
+
### What changed in the head
|
| 59 |
+
|
| 60 |
+
The pretrained PP-OCRv6 dictionary has **18 710** classes and no Cyrillic letters at all. Rows
|
| 61 |
+
**0 … 18 709** of the published dictionary are the pretrained ones (index 18 709 is the space);
|
| 62 |
+
rows **18 710 … 18 761** are the **52 Cyrillic characters** that were appended:
|
| 63 |
+
|
| 64 |
+
```
|
| 65 |
+
ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ + авдеикмнопрстцчшыья
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
The pretrained rows were **kept**, not replaced (`--expand-charset` grows the head and copies the
|
| 69 |
+
old rows). The extension was done in an earlier run of the same recipe; in this run the flag
|
| 70 |
+
reported «0 characters missing», i.e. the grown head was inherited from the previous checkpoint
|
| 71 |
+
together with its weights.
|
| 72 |
+
|
| 73 |
+
## The training run in numbers
|
| 74 |
|
| 75 |
| | |
|
| 76 |
|---|---|
|
| 77 |
+
| initialised from | `train/runs/ppocrv6/limbo_cyr3/best.pt` — the **«replay mix»** Cyrillic checkpoint (3 short Cyrillic types + digits + 5 Latin types, 4 epochs) |
|
| 78 |
+
| epochs × steps | 5 × 488 = **2 440** steps |
|
| 79 |
+
| batch size | 32 |
|
| 80 |
+
| learning rate | **8e-6** for the backbone **and** 8e-6 for the head |
|
| 81 |
+
| weight decay / grad clip | 0.01 / 1.0 |
|
| 82 |
+
| `--limit-per-type` | **1 500** |
|
| 83 |
+
| training examples | **15 594** |
|
| 84 |
+
| evaluation set | **11 400** images over 18 types |
|
| 85 |
+
| wall clock | **33.9 min** (2 032 s) at **108 img/s** (one GTX 1080 Ti, fp16/AMP) |
|
| 86 |
+
| seed | 42 · `--workers 0` (the sandbox this was trained in denies multiprocessing named pipes) |
|
| 87 |
|
| 88 |
+
### What was trained on, and what was not
|
| 89 |
|
| 90 |
+
**12 types were trained on.** Eight of them are fed as the **answer band** (the `roi` rectangle
|
| 91 |
+
of the dataset); the four that have no band at all are trained on the **whole frame**, so the
|
| 92 |
+
model has to find the code by itself:
|
|
|
|
| 93 |
|
| 94 |
+
| type | input |
|
| 95 |
|---|---|
|
| 96 |
+
| `out/Archy-X/Krypton/map` | band |
|
| 97 |
+
| `out/haq/mapcha/map` | band |
|
| 98 |
+
| `out/just-captcha/ProstoCaptcha/map` | band |
|
| 99 |
+
| `out/ringoame196-s-mcPlugin/ReproductionCAPTCHA/map` | band |
|
| 100 |
+
| `out_ru/GalleryCaptcha/ru-symbols` | band |
|
| 101 |
+
| `out_ru/GalleryCaptcha/ru-web` | band |
|
| 102 |
+
| `out_ru/LimboFilter/digits` | band |
|
| 103 |
+
| `out_ru/LimboFilter/symbols` | band |
|
| 104 |
+
| `out/InstantlyMoist/Captcha/textcaptcha` | **whole frame** |
|
| 105 |
+
| `out/LagBug/XProtect/code` | **whole frame** |
|
| 106 |
+
| `out/LagBug/XProtect/map` | **whole frame** |
|
| 107 |
+
| `out/irootdev/AnBot/captcha` | **whole frame** |
|
| 108 |
+
|
| 109 |
+
(The names above are the dataset's own `type_id` values.)
|
| 110 |
+
|
| 111 |
+
**6 types were never seen in training.** The unit of generalisation is the **plugin**, never the
|
| 112 |
+
picture: these six whole plugins were held out of the training set:
|
| 113 |
+
|
| 114 |
+
`out/JSH32/Minecaptcha/map`, `out/jonesdevelopment/sonar/standard`,
|
| 115 |
+
`out/jonesdevelopment/sonar/legacy`, `out/jonesdevelopment/sonar/complex`,
|
| 116 |
+
`out_ru/BotFilter/map`, `out_ru/CaptchaVerifyZ/map`.
|
| 117 |
+
|
| 118 |
+
**3 types were excluded on purpose:** `out_ru/GalleryCaptcha/ru-numberword`,
|
| 119 |
+
`out_ru/GalleryCaptcha/ru-numberword-caption` and `out_ru/LimboFilter/map`. In all three the picture shows a
|
| 120 |
+
Russian number **written out in words** while the player types **digits** (`text != answer` in
|
| 121 |
+
all 10 000 rows of each). Converting a written number into a typed one is a **solver step**,
|
| 122 |
+
not an OCR task, so the model was not asked to do it. That is a limitation of the *task*, not a
|
| 123 |
+
defect of the model — but it does mean these three types are **not supported**.
|
| 124 |
+
|
| 125 |
+
## Measured results
|
| 126 |
+
|
| 127 |
+
Measured inside the same run (`train/runs/ppocrv6/universal_v1/result.json`): 11 400 images over
|
| 128 |
+
18 types, greedy CTC decoding, exact match against the drawn string.
|
| 129 |
+
|
| 130 |
+
| aggregate | before this run¹ | **after** |
|
|
|
|
| 131 |
|---|---|---|
|
| 132 |
+
| median exact match over 18 types | 0.8750 | **0.8875** |
|
| 133 |
+
| median CER | 0.0394 | **0.0344** |
|
| 134 |
+
| micro exact match (11 400 images) | 0.6499 | **0.7797** |
|
| 135 |
+
| micro CER | 0.1795 | **0.0755** |
|
| 136 |
+
|
| 137 |
+
¹ «before this run» is the **zero step of this very run** — the checkpoint it was initialised
|
| 138 |
+
from (`limbo_cyr3/best.pt`), **not** the pretrained PP-OCRv6 base. The fine-tune moved the
|
| 139 |
+
median only a little (0.875 → 0.8875) but the **micro** average a lot (0.6499 → 0.7797): the
|
| 140 |
+
median is dominated by the types that were already solved, while the micro average counts every
|
| 141 |
+
image — and the frame-trained types that were unreadable rose sharply
|
| 142 |
+
(`out/LagBug/XProtect/code` 0.00 → 0.36, `out/LagBug/XProtect/map` 0.00 → 0.27,
|
| 143 |
+
`out/InstantlyMoist/Captcha/textcaptcha` 0.12 → 0.24).
|
| 144 |
+
|
| 145 |
+
### Types the model was trained on
|
| 146 |
+
|
| 147 |
+
These numbers are a **sample-level hold-out**: the first 200 rows of every type were taken out
|
| 148 |
+
of training, the rest were trained on. They are *not* unseen-plugin numbers.
|
| 149 |
+
|
| 150 |
+
| type | n | exact (zero step) | **exact** | CER |
|
| 151 |
+
|---|---:|---:|---:|---:|
|
| 152 |
+
| `out/Archy-X/Krypton/map` | 200 | 1.0000 | **1.0000** | 0.0000 |
|
| 153 |
+
| `out/ringoame196-s-mcPlugin/ReproductionCAPTCHA/map` | 200 | 1.0000 | **1.0000** | 0.0000 |
|
| 154 |
+
| `out/irootdev/AnBot/captcha` | 200 | 0.9350 | **0.9950** | 0.0017 |
|
| 155 |
+
| `out/haq/mapcha/map` | 200 | 0.9650 | 0.9550 | 0.0112 |
|
| 156 |
+
| `out_ru/GalleryCaptcha/ru-web` | 200 | 0.9300 | 0.9250 | 0.0150 |
|
| 157 |
+
| `out/just-captcha/ProstoCaptcha/map` | 200 | 0.9250 | 0.9150 | 0.0150 |
|
| 158 |
+
| `out_ru/LimboFilter/digits` | 200 | 0.8900 | 0.8900 | 0.0288 |
|
| 159 |
+
| `out_ru/GalleryCaptcha/ru-symbols` | 200 | 0.8600 | **0.8850** | 0.0400 |
|
| 160 |
+
| `out_ru/LimboFilter/symbols` | 200 | 0.7650 | **0.8000** | 0.0783 |
|
| 161 |
+
| `out/LagBug/XProtect/code` | 200 | 0.0000 | **0.3600** | 0.1542 |
|
| 162 |
+
| `out/LagBug/XProtect/map` | 200 | 0.0000 | **0.2700** | 0.1850 |
|
| 163 |
+
| `out/InstantlyMoist/Captcha/textcaptcha` | 200 | 0.1200 | **0.2400** | 0.4220 |
|
| 164 |
+
|
| 165 |
+
### Types the model never saw
|
| 166 |
+
|
| 167 |
+
Whole plugins that were held out of training — these are the honest generalisation numbers.
|
| 168 |
+
|
| 169 |
+
| type | n | exact (zero step) | **exact** | CER |
|
| 170 |
+
|---|---:|---:|---:|---:|
|
| 171 |
+
| `out_ru/CaptchaVerifyZ/map` | 1 500 | 0.9647 | **0.9780** | 0.0045 |
|
| 172 |
+
| `out/JSH32/Minecaptcha/map` | 1 500 | 0.9340 | **0.9373** | 0.0162 |
|
| 173 |
+
| `out_ru/BotFilter/map` | 1 500 | 0.7113 | **0.8140** | 0.0508 |
|
| 174 |
+
| `out/jonesdevelopment/sonar/legacy` | 1 500 | 0.5973 | **0.7693** | 0.0793 |
|
| 175 |
+
| `out/jonesdevelopment/sonar/complex` | 1 500 | 0.3520 | **0.6253** | 0.1293 |
|
| 176 |
+
| `out/jonesdevelopment/sonar/standard` | 1 500 | 0.2613 | **0.5707** | 0.1671 |
|
| 177 |
|
| 178 |
+
**One caveat about the protocol.** The evaluation runs on held-out rows of the **same generator
|
| 179 |
+
render** the model was trained on (the generators are not seeded, so a re-render is a different
|
| 180 |
+
sample set). For the twelve trained types the hold-out is by *sample*; only the six held-out
|
| 181 |
+
plugins above are genuinely unseen styles.
|
| 182 |
|
| 183 |
+
## No weight interpolation — and why the previous card said α = 0.4
|
| 184 |
+
|
| 185 |
+
The previous version of this repository published a WiSE-FT soup, `θ = 0.4·θ_ft + 0.6·θ_pretrained`.
|
| 186 |
+
That was the right call **for that model**: a small fine-tune on a few types degraded the unseen
|
| 187 |
+
plugins, and interpolating back towards the base restored them (measured there: base median
|
| 188 |
+
0.588 → α = 0.4 median 0.638 → α = 1.0 median 0.480).
|
| 189 |
+
|
| 190 |
+
This model is different in two ways: it is trained on **most** of the types it is evaluated on,
|
| 191 |
+
and its head has **grown**. On an expanded head the interpolation has nothing to interpolate
|
| 192 |
+
*towards* for the 52 appended rows — the base model has no such rows — while the encoder is
|
| 193 |
+
dragged back to the pretrained solution. Measured on the α-sweep of the `limbo_cyr3`
|
| 194 |
+
predecessor: the sweep's best α = 0.6 lifted the Latin held-out types from 0.644 to 0.772 and
|
| 195 |
+
at the same time **collapsed the Cyrillic ones**:
|
| 196 |
+
|
| 197 |
+
| type | α = 1.0 | after interpolation (α = 0.6) |
|
| 198 |
+
|---|---:|---:|
|
| 199 |
+
| `out_ru/LimboFilter/symbols` | 0.765 | **0.016** |
|
| 200 |
+
| `out_ru/GalleryCaptcha/ru-symbols` | 0.860 | **0.002** |
|
| 201 |
+
| `out_ru/LimboFilter/digits` | 0.890 | **0.666** |
|
| 202 |
+
|
| 203 |
+
So WiSE-FT is **destructive** on an expanded head. That soup was kept as an exhibit
|
| 204 |
+
(`train/runs/ppocrv6/limbo_cyr3/soup_best.pt`) and is **not** used here. What works for a
|
| 205 |
+
vocabulary expansion on this model is **replay** — keeping the old data in the training mix —
|
| 206 |
+
and this checkpoint is therefore published with **α = 1.0**: the fine-tuned weights, unchanged.
|
| 207 |
+
|
| 208 |
+
## What this model is NOT good at
|
| 209 |
+
|
| 210 |
+
* **It is not a universal captcha solver.** It is a fine-tune for a **known set of captcha
|
| 211 |
+
styles**: a plugin that draws in a style absent from the training set will not be handled.
|
| 212 |
+
For the six plugins that were held out, the honest expectation is the numbers in the table
|
| 213 |
+
above — between 0.57 and 0.98 exact match, not 1.0.
|
| 214 |
+
* **The three number-word types are not supported at all** (`out_ru/GalleryCaptcha/ru-numberword`,
|
| 215 |
+
`out_ru/GalleryCaptcha/ru-numberword-caption`, `out_ru/LimboFilter/map`): the picture shows a number
|
| 216 |
+
written in Russian words while the answer is digits. Use a solver step for those.
|
| 217 |
+
* **The weakest trained types are `out/LagBug/XProtect/code` (0.36) and `out/LagBug/XProtect/map`
|
| 218 |
+
(0.27)**: both are the same 130×130 sheet with per-character cumulative translation and 51
|
| 219 |
+
interference lines. `out/InstantlyMoist/Captcha/textcaptcha` (0.24) is a 128×128 sheet with
|
| 220 |
+
thin diagonal noise.
|
| 221 |
+
* **It reads a band, not a picture.** On a whole game screenshot it falls apart — see below.
|
| 222 |
|
| 223 |
## How to use it
|
| 224 |
|
|
|
|
| 227 |
from PIL import Image
|
| 228 |
from transformers import PPOCRV6SmallRecForTextRecognition, PPOCRV6SmallRecImageProcessor
|
| 229 |
|
| 230 |
+
REPO = "Titlehhhh/captchaocr-ppocrv6"
|
| 231 |
+
|
| 232 |
+
model = PPOCRV6SmallRecForTextRecognition.from_pretrained(REPO)
|
| 233 |
+
proc = PPOCRV6SmallRecImageProcessor.from_pretrained(REPO)
|
| 234 |
+
|
| 235 |
+
image = Image.open("001.png").convert("RGB") # a crop of the answer band
|
| 236 |
+
px = proc(images=[image], return_tensors="pt")["pixel_values"] # 48 x 320, mean = std = 0.5
|
| 237 |
|
|
|
|
|
|
|
| 238 |
with torch.no_grad():
|
| 239 |
logits = model.head.head(model.head.encoder(
|
| 240 |
model.model(px).last_hidden_state).last_hidden_state)
|
| 241 |
+
|
| 242 |
+
# greedy CTC decoding against character_list.txt (index 0 = "blank")
|
| 243 |
+
with open("character_list.txt", encoding="utf-8") as f:
|
| 244 |
+
dictionary = [line.rstrip("\r\n") for line in f] # rstrip, not strip: " " is a class
|
| 245 |
+
|
| 246 |
+
ids = logits.argmax(-1)[0].tolist()
|
| 247 |
+
text, prev = [], -1
|
| 248 |
+
for i in ids:
|
| 249 |
+
if i != prev and i != 0: # collapse repeats, drop the blank
|
| 250 |
+
text.append(dictionary[i])
|
| 251 |
+
prev = i
|
| 252 |
+
print("".join(text))
|
| 253 |
```
|
| 254 |
|
| 255 |
+
The same 18 762-character dictionary is also inside `preprocessor_config.json`, so the processor
|
| 256 |
+
can decode without the text file. The input geometry is 48 × 320 (resize + pad), and the
|
| 257 |
+
normalisation is `mean = std = 0.5` — the PaddleOCR convention the fine-tune was trained with.
|
| 258 |
+
**Do not let the processor fall back to the ImageNet statistics**: `preprocessor_config.json`
|
| 259 |
+
already carries the patched values.
|
| 260 |
+
|
| 261 |
+
### The important practical part: give it a band, not a screenshot
|
| 262 |
+
|
| 263 |
+
The model is trained on the **band** for 8 of its 12 types, so it expects a tight crop of the
|
| 264 |
+
code. On a **whole game screenshot** it collapses: measured on the six real screenshots of the
|
| 265 |
+
project owner, feeding the whole frame gives **2/6** correct readings.
|
| 266 |
+
|
| 267 |
+
The fix is a **cropper** — an «eye» in front of the recogniser. It proposes candidate text bands
|
| 268 |
+
(a per-type heuristic, an ink box, and a glyph detector), the model reads every candidate, and
|
| 269 |
+
the reading with the **highest confidence** wins (vote counts are only a tie-break). With the
|
| 270 |
+
cropper: **6/6** on those same six screenshots. The implementation ships with the project
|
| 271 |
+
(`webapp/cropper.py`, `webapp/app.py` — see
|
| 272 |
+
<https://github.com/Titlehhhh/CaptchaOCR> and `reports/CROPPER.md` for the measured numbers).
|
| 273 |
|
| 274 |
+
If you use the dataset: crop by `roi` whenever it is not `null`.
|
| 275 |
|
| 276 |
+
## The dataset this was trained on
|
| 277 |
+
|
| 278 |
+
[`Titlehhhh/captchaocr-dataset`](https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset)
|
| 279 |
+
— synthetic, every image is a re-render by this project's own Java generators of a captcha drawn
|
| 280 |
+
by an open-source Minecraft plugin. As published (commit
|
| 281 |
+
`dd4d7479a95f8e4c8ea19643dfe5889fe5dea113`):
|
| 282 |
+
|
| 283 |
+
| | |
|
| 284 |
+
|---|---|
|
| 285 |
+
| types | **20** (12 in `out`, 8 in `out_ru`), 13 plugins |
|
| 286 |
+
| samples | **200 000** (10 000 per type), 1.9 GiB of PNG bytes |
|
| 287 |
+
| files | 9 Parquet shards (`data/train-*-of-00009.parquet`), one row per sample, raw PNG in `image` |
|
| 288 |
+
| bands | 12 types carry an `roi` band; 119 984 rows have one, 80 016 do not |
|
| 289 |
+
| `text != answer` | 3 types (the Russian number words) |
|
| 290 |
+
| alphabet | 115 distinct characters over the whole set |
|
| 291 |
+
|
| 292 |
+
Two facts worth stating plainly:
|
| 293 |
+
|
| 294 |
+
* The model was trained on the **project's own render** of that dataset (10 000 images per
|
| 295 |
+
type), not on the published Parquet shards — the pixels are the same style, the sample set is
|
| 296 |
+
that of the local render.
|
| 297 |
+
* The type **`out_ru/LimboFilter/digits`** (four digits, real-server style) was trained on and
|
| 298 |
+
**is not part of the published dataset snapshot**: the generator gained that type after the
|
| 299 |
+
dataset was exported.
|
| 300 |
+
|
| 301 |
+
### Provenance and licence
|
| 302 |
+
|
| 303 |
+
* Base model: `PaddlePaddle/PP-OCRv6_medium_rec_safetensors`, **Apache-2.0**. These weights are
|
| 304 |
+
a derivative of it.
|
| 305 |
+
* The dataset is **synthetic** — re-renders, not screenshots of a live server; no picture, page
|
| 306 |
+
or answer of a third-party commercial service is used or published.
|
| 307 |
+
* The dataset is published under **the licence of the upstream plugin each row came from**
|
| 308 |
+
(`license_name: mixed-per-plugin-licences`; the per-plugin breakdown is in the dataset
|
| 309 |
+
repository's `LICENSE` and in its card). Some of those licences forbid commercial hosting or
|
| 310 |
+
reserve all rights — **read them before you use the data**.
|
| 311 |
+
* The 52 characters added to the head are the Cyrillic alphabet; the images that taught them
|
| 312 |
+
come from the `GalleryCaptcha` and `LimboFilter` types, whose licences are listed above.
|
| 313 |
|
| 314 |
## Reproducing
|
| 315 |
|
| 316 |
```powershell
|
| 317 |
+
$env:PYTHONIOENCODING="utf-8"
|
| 318 |
+
$env:HF_HOME="C:\Users\Title\Projects\CapthaModel\.hf-cache"
|
| 319 |
+
|
| 320 |
+
# the run that produced these weights
|
| 321 |
+
.venv-gpu\Scripts\python.exe train\finetune_ppocr.py --out train\runs\ppocrv6\universal_v1 `
|
| 322 |
+
--init-from train\runs\ppocrv6\limbo_cyr3\best.pt --expand-charset --train-frames `
|
| 323 |
+
--epochs 5 --batch-size 32 --lr 8e-6 --head-lr 8e-6 --workers 0 --seed 42 `
|
| 324 |
+
--limit-per-type 1500 --holdout-per-type 200 --eval-every 5 --save-every 400 --max-minutes 60 `
|
| 325 |
+
--val-types out/jonesdevelopment/sonar/standard out/jonesdevelopment/sonar/legacy out/jonesdevelopment/sonar/complex out_ru/BotFilter/map out_ru/CaptchaVerifyZ/map out/laforetbrut/are-you-sure/code out/JSH32/Minecaptcha/map `
|
| 326 |
+
--exclude-types ru-numberword ru-numberword-caption LimboFilter/map
|
| 327 |
+
|
| 328 |
+
# the export (alpha = 1.0 -> the checkpoint itself, no interpolation)
|
| 329 |
+
.venv-gpu\Scripts\python.exe train\export_ppocr_hf.py `
|
| 330 |
+
--ckpt train\runs\ppocrv6\universal_v1\best.pt --soup-alpha 1.0 `
|
| 331 |
+
--out train\runs\ppocrv6\universal_v1\export_hf
|
| 332 |
```
|
| 333 |
+
|
| 334 |
+
Notes on the command as it was actually run:
|
| 335 |
+
|
| 336 |
+
* `--init-from train\runs\ppocrv6\limbo_cyr3\best.pt` points at the **replay-mix** checkpoint of
|
| 337 |
+
the same recipe (the chain is `limbo_cyr` → charset expansion → `limbo_cyr3` → replay mix →
|
| 338 |
+
`universal_v1`); the whole chain is documented in `reports/TRAINING_LIMBO_DIGITS.md` §8.
|
| 339 |
+
* The filter `out/laforetbrut/are-you-sure/code` matched **no type** in the dataset and had no
|
| 340 |
+
effect; the other six `--val-types` are the six held-out plugins.
|
| 341 |
+
* `--expand-charset` was a **no-op in this run** («0 characters missing»): the head had already
|
| 342 |
+
been grown to 18 762 classes by the predecessor checkpoint.
|
| 343 |
+
* The dataset is **not seeded**: a re-render produces a different sample set, so the exact
|
| 344 |
+
per-type numbers above belong to the render this checkpoint was measured on.
|