Titlehhhh commited on
Commit
93fab11
·
verified ·
1 Parent(s): 37556a8

Rewrite model card: measured results, held-out table, limitations

Browse files
Files changed (1) hide show
  1. README.md +127 -292
README.md CHANGED
@@ -18,232 +18,45 @@ base_model:
18
  - PaddlePaddle/PP-OCRv6_medium_rec_safetensors
19
  ---
20
 
21
- # CaptchaOCR PP-OCRv6 — a text recogniser for Minecraft anti-bot captchas
22
 
23
- A fine-tune of **PP-OCRv6_medium_rec** on Minecraft anti-bot captchas whose CTC head was
24
- **extended to 18 762 classes**, so that it reads Cyrillic as well as Latin letters and digits.
25
- Feed it **one crop of the answer band**, get the string drawn on it.
26
 
27
- > **This card describes the `universal_v1` checkpoint — the weights that are actually in this
28
- > repository.** The card that was here before described an older run (`all20`) whose weights are
29
- > **no longer** published: 18 710 classes, 15 types, median exact 0.920. Those numbers were true
30
- > of that model and are **not** true of this one.
31
 
32
- ## What is in this repository
 
33
 
34
- | file | what it is |
35
- |---|---|
36
- | `model.safetensors` | the weights: 308 tensors, 19 186 154 float parameters |
37
- | `config.json` | the base architecture, with `head_out_channels` patched to 18 762 |
38
- | `preprocessor_config.json` | the base processor + the PaddleOCR normalisation + the 18 762-character dictionary |
39
- | `processor_config.json` | `PPOCRV6SmallRecImageProcessor` |
40
- | `character_list.txt` | the dictionary, one entry per line, index 0 = `blank` |
41
- | `finetune_meta.json` | the provenance of the export (checkpoint, epoch, step, α) |
42
- | `README.md` | this card |
43
-
44
- Provenance, as shipped in `finetune_meta.json`:
45
-
46
- | | |
47
- |---|---|
48
- | base model | `PaddlePaddle/PP-OCRv6_medium_rec_safetensors` (Apache-2.0) |
49
- | fine-tuned checkpoint | `train/runs/ppocrv6/universal_v1/best.pt` |
50
- | epoch / step | 4 (the last of 5) / 2 440 |
51
- | classes | **18 762** (index 0 = `blank`) — the pretrained head is **kept and extended**, not replaced |
52
- | float parameters | **19 186 154** |
53
- | weight interpolation | **none** — α = 1.0, the checkpoint exactly (see below) |
54
- | transformers class | `PPOCRV6SmallRecForTextRecognition` |
55
- | normalisation | mean = std = 0.5 (the PaddleOCR convention) |
56
- | exported by | `train/export_ppocr_hf.py` |
57
-
58
- ### What changed in the head
59
-
60
- The pretrained PP-OCRv6 dictionary has **18 710** classes and no Cyrillic letters at all. Rows
61
- **0 … 18 709** of the published dictionary are the pretrained ones (index 18 709 is the space);
62
- rows **18 710 … 18 761** are the **52 Cyrillic characters** that were appended:
63
-
64
- ```
65
- ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ + авдеикмнопрстцчшыья
66
- ```
67
-
68
- The pretrained rows were **kept**, not replaced (`--expand-charset` grows the head and copies the
69
- old rows). The extension was done in an earlier run of the same recipe; in this run the flag
70
- reported «0 characters missing», i.e. the grown head was inherited from the previous checkpoint
71
- together with its weights.
72
-
73
- ## The training run in numbers
74
-
75
- | | |
76
- |---|---|
77
- | initialised from | `train/runs/ppocrv6/limbo_cyr3/best.pt` — the **«replay mix»** Cyrillic checkpoint (3 short Cyrillic types + digits + 5 Latin types, 4 epochs) |
78
- | epochs × steps | 5 × 488 = **2 440** steps |
79
- | batch size | 32 |
80
- | learning rate | **8e-6** for the backbone **and** 8e-6 for the head |
81
- | weight decay / grad clip | 0.01 / 1.0 |
82
- | `--limit-per-type` | **1 500** |
83
- | training examples | **15 594** |
84
- | evaluation set | **11 400** images over 18 types |
85
- | wall clock | **33.9 min** (2 032 s) at **108 img/s** (one GTX 1080 Ti, fp16/AMP) |
86
- | seed | 42 · `--workers 0` (the sandbox this was trained in denies multiprocessing named pipes) |
87
-
88
- ### What was trained on, and what was not
89
-
90
- **12 types were trained on.** Eight of them are fed as the **answer band** (the `roi` rectangle
91
- of the dataset); the four that have no band at all are trained on the **whole frame**, so the
92
- model has to find the code by itself:
93
-
94
- | type | input |
95
- |---|---|
96
- | `out/Archy-X/Krypton/map` | band |
97
- | `out/haq/mapcha/map` | band |
98
- | `out/just-captcha/ProstoCaptcha/map` | band |
99
- | `out/ringoame196-s-mcPlugin/ReproductionCAPTCHA/map` | band |
100
- | `out_ru/GalleryCaptcha/ru-symbols` | band |
101
- | `out_ru/GalleryCaptcha/ru-web` | band |
102
- | `out_ru/LimboFilter/digits` | band |
103
- | `out_ru/LimboFilter/symbols` | band |
104
- | `out/InstantlyMoist/Captcha/textcaptcha` | **whole frame** |
105
- | `out/LagBug/XProtect/code` | **whole frame** |
106
- | `out/LagBug/XProtect/map` | **whole frame** |
107
- | `out/irootdev/AnBot/captcha` | **whole frame** |
108
-
109
- (The names above are the dataset's own `type_id` values.)
110
-
111
- **6 types were never seen in training.** The unit of generalisation is the **plugin**, never the
112
- picture: these six whole plugins were held out of the training set:
113
-
114
- `out/JSH32/Minecaptcha/map`, `out/jonesdevelopment/sonar/standard`,
115
- `out/jonesdevelopment/sonar/legacy`, `out/jonesdevelopment/sonar/complex`,
116
- `out_ru/BotFilter/map`, `out_ru/CaptchaVerifyZ/map`.
117
-
118
- **3 types were excluded on purpose:** `out_ru/GalleryCaptcha/ru-numberword`,
119
- `out_ru/GalleryCaptcha/ru-numberword-caption` and `out_ru/LimboFilter/map`. In all three the picture shows a
120
- Russian number **written out in words** while the player types **digits** (`text != answer` in
121
- all 10 000 rows of each). Converting a written number into a typed one is a **solver step**,
122
- not an OCR task, so the model was not asked to do it. That is a limitation of the *task*, not a
123
- defect of the model — but it does mean these three types are **not supported**.
124
-
125
- ## Measured results
126
-
127
- Measured inside the same run (`train/runs/ppocrv6/universal_v1/result.json`): 11 400 images over
128
- 18 types, greedy CTC decoding, exact match against the drawn string.
129
-
130
- | aggregate | before this run¹ | **after** |
131
- |---|---|---|
132
- | median exact match over 18 types | 0.8750 | **0.8875** |
133
- | median CER | 0.0394 | **0.0344** |
134
- | micro exact match (11 400 images) | 0.6499 | **0.7797** |
135
- | micro CER | 0.1795 | **0.0755** |
136
-
137
- ¹ «before this run» is the **zero step of this very run** — the checkpoint it was initialised
138
- from (`limbo_cyr3/best.pt`), **not** the pretrained PP-OCRv6 base. The fine-tune moved the
139
- median only a little (0.875 → 0.8875) but the **micro** average a lot (0.6499 → 0.7797): the
140
- median is dominated by the types that were already solved, while the micro average counts every
141
- image — and the frame-trained types that were unreadable rose sharply
142
- (`out/LagBug/XProtect/code` 0.00 → 0.36, `out/LagBug/XProtect/map` 0.00 → 0.27,
143
- `out/InstantlyMoist/Captcha/textcaptcha` 0.12 → 0.24).
144
-
145
- ### Types the model was trained on
146
-
147
- These numbers are a **sample-level hold-out**: the first 200 rows of every type were taken out
148
- of training, the rest were trained on. They are *not* unseen-plugin numbers.
149
-
150
- | type | n | exact (zero step) | **exact** | CER |
151
- |---|---:|---:|---:|---:|
152
- | `out/Archy-X/Krypton/map` | 200 | 1.0000 | **1.0000** | 0.0000 |
153
- | `out/ringoame196-s-mcPlugin/ReproductionCAPTCHA/map` | 200 | 1.0000 | **1.0000** | 0.0000 |
154
- | `out/irootdev/AnBot/captcha` | 200 | 0.9350 | **0.9950** | 0.0017 |
155
- | `out/haq/mapcha/map` | 200 | 0.9650 | 0.9550 | 0.0112 |
156
- | `out_ru/GalleryCaptcha/ru-web` | 200 | 0.9300 | 0.9250 | 0.0150 |
157
- | `out/just-captcha/ProstoCaptcha/map` | 200 | 0.9250 | 0.9150 | 0.0150 |
158
- | `out_ru/LimboFilter/digits` | 200 | 0.8900 | 0.8900 | 0.0288 |
159
- | `out_ru/GalleryCaptcha/ru-symbols` | 200 | 0.8600 | **0.8850** | 0.0400 |
160
- | `out_ru/LimboFilter/symbols` | 200 | 0.7650 | **0.8000** | 0.0783 |
161
- | `out/LagBug/XProtect/code` | 200 | 0.0000 | **0.3600** | 0.1542 |
162
- | `out/LagBug/XProtect/map` | 200 | 0.0000 | **0.2700** | 0.1850 |
163
- | `out/InstantlyMoist/Captcha/textcaptcha` | 200 | 0.1200 | **0.2400** | 0.4220 |
164
-
165
- ### Types the model never saw
166
-
167
- Whole plugins that were held out of training — these are the honest generalisation numbers.
168
-
169
- | type | n | exact (zero step) | **exact** | CER |
170
- |---|---:|---:|---:|---:|
171
- | `out_ru/CaptchaVerifyZ/map` | 1 500 | 0.9647 | **0.9780** | 0.0045 |
172
- | `out/JSH32/Minecaptcha/map` | 1 500 | 0.9340 | **0.9373** | 0.0162 |
173
- | `out_ru/BotFilter/map` | 1 500 | 0.7113 | **0.8140** | 0.0508 |
174
- | `out/jonesdevelopment/sonar/legacy` | 1 500 | 0.5973 | **0.7693** | 0.0793 |
175
- | `out/jonesdevelopment/sonar/complex` | 1 500 | 0.3520 | **0.6253** | 0.1293 |
176
- | `out/jonesdevelopment/sonar/standard` | 1 500 | 0.2613 | **0.5707** | 0.1671 |
177
-
178
- **One caveat about the protocol.** The evaluation runs on held-out rows of the **same generator
179
- render** the model was trained on (the generators are not seeded, so a re-render is a different
180
- sample set). For the twelve trained types the hold-out is by *sample*; only the six held-out
181
- plugins above are genuinely unseen styles.
182
-
183
- ## No weight interpolation — and why the previous card said α = 0.4
184
-
185
- The previous version of this repository published a WiSE-FT soup, `θ = 0.4·θ_ft + 0.6·θ_pretrained`.
186
- That was the right call **for that model**: a small fine-tune on a few types degraded the unseen
187
- plugins, and interpolating back towards the base restored them (measured there: base median
188
- 0.588 → α = 0.4 median 0.638 → α = 1.0 median 0.480).
189
-
190
- This model is different in two ways: it is trained on **most** of the types it is evaluated on,
191
- and its head has **grown**. On an expanded head the interpolation has nothing to interpolate
192
- *towards* for the 52 appended rows — the base model has no such rows — while the encoder is
193
- dragged back to the pretrained solution. Measured on the α-sweep of the `limbo_cyr3`
194
- predecessor: the sweep's best α = 0.6 lifted the Latin held-out types from 0.644 to 0.772 and
195
- at the same time **collapsed the Cyrillic ones**:
196
-
197
- | type | α = 1.0 | after interpolation (α = 0.6) |
198
- |---|---:|---:|
199
- | `out_ru/LimboFilter/symbols` | 0.765 | **0.016** |
200
- | `out_ru/GalleryCaptcha/ru-symbols` | 0.860 | **0.002** |
201
- | `out_ru/LimboFilter/digits` | 0.890 | **0.666** |
202
-
203
- So WiSE-FT is **destructive** on an expanded head. That soup was kept as an exhibit
204
- (`train/runs/ppocrv6/limbo_cyr3/soup_best.pt`) and is **not** used here. What works for a
205
- vocabulary expansion on this model is **replay** — keeping the old data in the training mix —
206
- and this checkpoint is therefore published with **α = 1.0**: the fine-tuned weights, unchanged.
207
-
208
- ## What this model is NOT good at
209
-
210
- * **It is not a universal captcha solver.** It is a fine-tune for a **known set of captcha
211
- styles**: a plugin that draws in a style absent from the training set will not be handled.
212
- For the six plugins that were held out, the honest expectation is the numbers in the table
213
- above — between 0.57 and 0.98 exact match, not 1.0.
214
- * **The three number-word types are not supported at all** (`out_ru/GalleryCaptcha/ru-numberword`,
215
- `out_ru/GalleryCaptcha/ru-numberword-caption`, `out_ru/LimboFilter/map`): the picture shows a number
216
- written in Russian words while the answer is digits. Use a solver step for those.
217
- * **The weakest trained types are `out/LagBug/XProtect/code` (0.36) and `out/LagBug/XProtect/map`
218
- (0.27)**: both are the same 130×130 sheet with per-character cumulative translation and 51
219
- interference lines. `out/InstantlyMoist/Captcha/textcaptcha` (0.24) is a 128×128 sheet with
220
- thin diagonal noise.
221
- * **It reads a band, not a picture.** On a whole game screenshot it falls apart — see below.
222
-
223
- ## How to use it
224
 
225
  ```python
226
  import torch
227
  from PIL import Image
228
- from transformers import PPOCRV6SmallRecForTextRecognition, PPOCRV6SmallRecImageProcessor
 
229
 
230
  REPO = "Titlehhhh/captchaocr-ppocrv6"
231
-
232
- model = PPOCRV6SmallRecForTextRecognition.from_pretrained(REPO)
233
  proc = PPOCRV6SmallRecImageProcessor.from_pretrained(REPO)
 
 
 
 
 
234
 
235
- image = Image.open("001.png").convert("RGB") # a crop of the answer band
236
- px = proc(images=[image], return_tensors="pt")["pixel_values"] # 48 x 320, mean = std = 0.5
 
237
 
238
- with torch.no_grad():
239
- logits = model.head.head(model.head.encoder(
240
- model.model(px).last_hidden_state).last_hidden_state)
 
241
 
242
- # greedy CTC decoding against character_list.txt (index 0 = "blank")
243
  with open("character_list.txt", encoding="utf-8") as f:
244
  dictionary = [line.rstrip("\r\n") for line in f] # rstrip, not strip: " " is a class
245
-
246
- ids = logits.argmax(-1)[0].tolist()
247
  text, prev = [], -1
248
  for i in ids:
249
  if i != prev and i != 0: # collapse repeats, drop the blank
@@ -251,94 +64,116 @@ for i in ids:
251
  prev = i
252
  print("".join(text))
253
  ```
 
254
 
255
- The same 18 762-character dictionary is also inside `preprocessor_config.json`, so the processor
256
- can decode without the text file. The input geometry is 48 × 320 (resize + pad), and the
257
- normalisation is `mean = std = 0.5` — the PaddleOCR convention the fine-tune was trained with.
258
- **Do not let the processor fall back to the ImageNet statistics**: `preprocessor_config.json`
259
- already carries the patched values.
260
 
261
- ### The important practical part: give it a band, not a screenshot
 
 
 
 
 
262
 
263
- The model is trained on the **band** for 8 of its 12 types, so it expects a tight crop of the
264
- code. On a **whole game screenshot** it collapses: measured on the six real screenshots of the
265
- project owner, feeding the whole frame gives **2/6** correct readings.
266
 
267
- The fix is a **cropper** — an «eye» in front of the recogniser. It proposes candidate text bands
268
- (a per-type heuristic, an ink box, and a glyph detector), the model reads every candidate, and
269
- the reading with the **highest confidence** wins (vote counts are only a tie-break). With the
270
- cropper: **6/6** on those same six screenshots. The implementation ships with the project
271
- (`webapp/cropper.py`, `webapp/app.py` — see
272
- <https://github.com/Titlehhhh/CaptchaOCR> and `reports/CROPPER.md` for the measured numbers).
273
 
274
- If you use the dataset: crop by `roi` whenever it is not `null`.
 
 
 
 
 
275
 
276
- ## The dataset this was trained on
277
 
278
- [`Titlehhhh/captchaocr-dataset`](https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset)
279
- — synthetic, every image is a re-render by this project's own Java generators of a captcha drawn
280
- by an open-source Minecraft plugin. As published (commit
281
- `dd4d7479a95f8e4c8ea19643dfe5889fe5dea113`):
 
 
 
 
282
 
283
- | | |
284
- |---|---|
285
- | types | **20** (12 in `out`, 8 in `out_ru`), 13 plugins |
286
- | samples | **200 000** (10 000 per type), 1.9 GiB of PNG bytes |
287
- | files | 9 Parquet shards (`data/train-*-of-00009.parquet`), one row per sample, raw PNG in `image` |
288
- | bands | 12 types carry an `roi` band; 119 984 rows have one, 80 016 do not |
289
- | `text != answer` | 3 types (the Russian number words) |
290
- | alphabet | 115 distinct characters over the whole set |
291
-
292
- Two facts worth stating plainly:
293
-
294
- * The model was trained on the **project's own render** of that dataset (10 000 images per
295
- type), not on the published Parquet shards — the pixels are the same style, the sample set is
296
- that of the local render.
297
- * The type **`out_ru/LimboFilter/digits`** (four digits, real-server style) was trained on and
298
- **is not part of the published dataset snapshot**: the generator gained that type after the
299
- dataset was exported.
300
-
301
- ### Provenance and licence
302
-
303
- * Base model: `PaddlePaddle/PP-OCRv6_medium_rec_safetensors`, **Apache-2.0**. These weights are
304
- a derivative of it.
305
- * The dataset is **synthetic** — re-renders, not screenshots of a live server; no picture, page
306
- or answer of a third-party commercial service is used or published.
307
- * The dataset is published under **the licence of the upstream plugin each row came from**
308
- (`license_name: mixed-per-plugin-licences`; the per-plugin breakdown is in the dataset
309
- repository's `LICENSE` and in its card). Some of those licences forbid commercial hosting or
310
- reserve all rights — **read them before you use the data**.
311
- * The 52 characters added to the head are the Cyrillic alphabet; the images that taught them
312
- come from the `GalleryCaptcha` and `LimboFilter` types, whose licences are listed above.
313
-
314
- ## Reproducing
315
-
316
- ```powershell
317
- $env:PYTHONIOENCODING="utf-8"
318
- $env:HF_HOME="C:\Users\Title\Projects\CapthaModel\.hf-cache"
319
-
320
- # the run that produced these weights
321
- .venv-gpu\Scripts\python.exe train\finetune_ppocr.py --out train\runs\ppocrv6\universal_v1 `
322
- --init-from train\runs\ppocrv6\limbo_cyr3\best.pt --expand-charset --train-frames `
323
- --epochs 5 --batch-size 32 --lr 8e-6 --head-lr 8e-6 --workers 0 --seed 42 `
324
- --limit-per-type 1500 --holdout-per-type 200 --eval-every 5 --save-every 400 --max-minutes 60 `
325
- --val-types out/jonesdevelopment/sonar/standard out/jonesdevelopment/sonar/legacy out/jonesdevelopment/sonar/complex out_ru/BotFilter/map out_ru/CaptchaVerifyZ/map out/laforetbrut/are-you-sure/code out/JSH32/Minecaptcha/map `
326
- --exclude-types ru-numberword ru-numberword-caption LimboFilter/map
327
-
328
- # the export (alpha = 1.0 -> the checkpoint itself, no interpolation)
329
- .venv-gpu\Scripts\python.exe train\export_ppocr_hf.py `
330
- --ckpt train\runs\ppocrv6\universal_v1\best.pt --soup-alpha 1.0 `
331
- --out train\runs\ppocrv6\universal_v1\export_hf
332
- ```
333
 
334
- Notes on the command as it was actually run:
335
-
336
- * `--init-from train\runs\ppocrv6\limbo_cyr3\best.pt` points at the **replay-mix** checkpoint of
337
- the same recipe (the chain is `limbo_cyr` → charset expansion → `limbo_cyr3` → replay mix →
338
- `universal_v1`); the whole chain is documented in `reports/TRAINING_LIMBO_DIGITS.md` §8.
339
- * The filter `out/laforetbrut/are-you-sure/code` matched **no type** in the dataset and had no
340
- effect; the other six `--val-types` are the six held-out plugins.
341
- * `--expand-charset` was a **no-op in this run** («0 characters missing»): the head had already
342
- been grown to 18 762 classes by the predecessor checkpoint.
343
- * The dataset is **not seeded**: a re-render produces a different sample set, so the exact
344
- per-type numbers above belong to the render this checkpoint was measured on.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
  - PaddlePaddle/PP-OCRv6_medium_rec_safetensors
19
  ---
20
 
21
+ <img src="https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset/resolve/main/assets/model_hero.png" alt="A haq/mapcha/map captcha frame, the answer band cropped from it by roi, and the string the model decodes from that band: ly8r" width="100%">
22
 
23
+ ![base](https://img.shields.io/badge/base-PP--OCRv6_medium_rec-0b5cad) ![params](https://img.shields.io/badge/params-19.2M-informational) ![classes](https://img.shields.io/badge/CTC_classes-18%2C762-blueviolet) ![median](https://img.shields.io/badge/median_exact-88.75%25-brightgreen) ![cer](https://img.shields.io/badge/median_CER-3.44%25-green) ![heldout](https://img.shields.io/badge/held--out_plugins-6-yellow) ![license](https://img.shields.io/badge/license-Apache--2.0-blue) ![task](https://img.shields.io/badge/task-image--to--text-success)
 
 
24
 
25
+ # CaptchaOCR PP-OCRv6 — captcha text recogniser
 
 
 
26
 
27
+ A PP-OCRv6 fine-tune that reads a captcha image, usually the answer band: 18 evaluated styles,
28
+ Latin letters, digits, and part of the Cyrillic alphabet.
29
 
30
+ ## Use it in 10 lines
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
  ```python
33
  import torch
34
  from PIL import Image
35
+ from transformers import PPOCRV6SmallRecForTextRecognition
36
+ from transformers import PPOCRV6SmallRecImageProcessor
37
 
38
  REPO = "Titlehhhh/captchaocr-ppocrv6"
39
+ model = PPOCRV6SmallRecForTextRecognition.from_pretrained(REPO).eval()
 
40
  proc = PPOCRV6SmallRecImageProcessor.from_pretrained(REPO)
41
+ inputs = proc(images=[Image.open("band.png").convert("RGB")], return_tensors="pt")
42
+ with torch.no_grad():
43
+ logits = model(**inputs)
44
+ print(proc.post_process_text_recognition(logits)) # [{'text': 'ly8r', 'score': 0.96}]
45
+ ```
46
 
47
+ `band.png` is a tight crop of the answer band, not a screenshot. The processor resizes it to
48
+ 48 × 320, normalises it, and decodes CTC with the 18 762-entry dictionary from
49
+ `preprocessor_config.json`. This exact snippet was run against these weights.
50
 
51
+ <details>
52
+ <summary>Manual CTC decoding</summary>
53
+
54
+ If you would rather decode by hand, the forward returns the logits in `last_hidden_state`:
55
 
56
+ ```python
57
  with open("character_list.txt", encoding="utf-8") as f:
58
  dictionary = [line.rstrip("\r\n") for line in f] # rstrip, not strip: " " is a class
59
+ ids = logits.last_hidden_state.argmax(-1)[0].tolist()
 
60
  text, prev = [], -1
61
  for i in ids:
62
  if i != prev and i != 0: # collapse repeats, drop the blank
 
64
  prev = i
65
  print("".join(text))
66
  ```
67
+ </details>
68
 
69
+ ## What it supports
 
 
 
 
70
 
71
+ | Supports | Does not support |
72
+ |---|---|
73
+ | **18 evaluated styles**: 11 from the published dataset snapshot, 1 unpublished (`LimboFilter/digits`), 6 held-out plugins | **3 types that draw a number as a word** while the answer is digits: `ru-numberword`, `ru-numberword-caption`, `LimboFilter/map`. Those need a solver step, not OCR |
74
+ | Latin letters, digits, and 52 of 66 Cyrillic letters: all 33 uppercase, 19 lowercase | 14 lowercase Cyrillic letters are not classes: б, г, ё, ж, з, й, л, у, ф, х, щ, ъ, э, ю. Characters outside `character_list.txt` are never produced |
75
+ | A tight crop of the answer band (`roi` in the dataset), noisy bands with lines, per-character shifts, coloured backgrounds | A whole game screenshot without a cropper: 2 of 6 screenshots read correctly as-is, 6 of 6 with the project's cropper |
76
+ | Styles close to the trained plugins, including other plugins of the same visual family | A style absent from training: expect 0.57–0.98 exact, never 1.0 |
77
 
78
+ ## Results
 
 
79
 
80
+ Exact match and CER after greedy CTC decoding, on 11 400 images of 18 styles: 200 rows per trained
81
+ style plus 1 500 rows per held-out plugin. The median is taken over the 18 per-style scores; the
82
+ micro average counts every image, and 9 000 of the 11 400 belong to held-out plugins.
 
 
 
83
 
84
+ | Metric | Score |
85
+ |---|---:|
86
+ | median exact match (18 styles) | **0.8875** |
87
+ | median CER | **0.0344** |
88
+ | micro exact match (11 400 images) | **0.7797** |
89
+ | micro CER | **0.0755** |
90
 
91
+ **Plugins never seen in training** — what to expect on a new style:
92
 
93
+ | Plugin | Style | Exact | CER |
94
+ |---|---|---:|---:|
95
+ | CaptchaVerifyZ | map | 0.9780 | 0.0045 |
96
+ | Minecaptcha | map | 0.9373 | 0.0162 |
97
+ | BotFilter | map | 0.8140 | 0.0508 |
98
+ | sonar | legacy | 0.7693 | 0.0793 |
99
+ | sonar | complex | 0.6253 | 0.1293 |
100
+ | sonar | standard | 0.5707 | 0.1671 |
101
 
102
+ <details>
103
+ <summary>Per-style results on the trained styles</summary>
104
+
105
+ Sample-level hold-out: the first 200 rows of each style were taken out of training.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
106
 
107
+ | Style | Exact | CER |
108
+ |---|---:|---:|
109
+ | `Archy-X/Krypton/map` | 1.0000 | 0.0000 |
110
+ | `ringoame196-s-mcPlugin/ReproductionCAPTCHA/map` | 1.0000 | 0.0000 |
111
+ | `irootdev/AnBot/captcha` | 0.9950 | 0.0017 |
112
+ | `haq/mapcha/map` | 0.9550 | 0.0112 |
113
+ | `GalleryCaptcha/ru-web` | 0.9250 | 0.0150 |
114
+ | `just-captcha/ProstoCaptcha/map` | 0.9150 | 0.0150 |
115
+ | `LimboFilter/digits` (unpublished) | 0.8900 | 0.0288 |
116
+ | `GalleryCaptcha/ru-symbols` | 0.8850 | 0.0400 |
117
+ | `LimboFilter/symbols` | 0.8000 | 0.0783 |
118
+ | `LagBug/XProtect/code` | 0.3600 | 0.1542 |
119
+ | `LagBug/XProtect/map` | 0.2700 | 0.1850 |
120
+ | `InstantlyMoist/Captcha/textcaptcha` | 0.2400 | 0.4220 |
121
+
122
+ </details>
123
+
124
+ ## Limitations
125
+
126
+ * **Four of the twelve trained styles are fed the whole frame**, not a band. Three of them are the
127
+ weakest: `XProtect/code` 0.36, `XProtect/map` 0.27, `textcaptcha` 0.24. The fourth,
128
+ `AnBot/captcha`, scores 0.9950 on the frame — the input is not what makes a style hard.
129
+ * **The median is carried by the easy styles.** Seven of the twelve trained styles score 0.89 or
130
+ better. On a style unlike the training set, read the held-out table instead: 0.57–0.98.
131
+ * **Screenshots need a cropper.** Feed a raw game frame and 2 of 6 real screenshots are read
132
+ correctly; with the cropper, 6 of 6. The cropper ships with the code, not with these weights.
133
+ * **The micro average is mostly held-out data** (9 000 of 11 400 images), so it answers "how does
134
+ it do on unseen plugins", not "accuracy in general".
135
+ * **It is not an API and not a solver for new plugins.** One image in, one string out; a style
136
+ absent from training has no guarantee.
137
+
138
+ ## Dataset
139
+
140
+ [`Titlehhhh/captchaocr-dataset`](https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset) is
141
+ synthetic: every image is a re-render by this project's Java generators of a captcha drawn by an
142
+ open-source Minecraft plugin. The published snapshot holds 20 styles and 200 000 samples. A 21st
143
+ style, `out_ru/LimboFilter/digits`, was generated after the export and is not in the snapshot —
144
+ which is why 18 evaluated styles = 11 from the snapshot + 1 unpublished + 6 held out.
145
+
146
+ <details>
147
+ <summary>Training</summary>
148
+
149
+ * Base: `PP-OCRv6_medium_rec` (Apache-2.0). The CTC head was kept and grown from 18 710 to
150
+ 18 762 classes: rows 18 710–18 761 are 52 appended Cyrillic characters.
151
+ * Initialised from an earlier Cyrillic run of the same recipe that replayed Latin data, then
152
+ fine-tuned on 12 styles.
153
+ * 5 epochs × 488 steps = 2 440 steps, batch 32, learning rate 8e-6 for backbone and head,
154
+ weight decay 0.01, gradient clipping 1.0.
155
+ * 15 594 training examples: 1 300 per style after a 200-row hold-out, from a 10 000-per-style
156
+ render (the run used `--limit-per-type 1500`).
157
+ * 33.9 min on one GTX 1080 Ti, 108 images/s, fp16.
158
+ * Whole-frame styles were trained with the frame as input; the rest with the `roi` band.
159
+ * Full command line and scripts:
160
+ [github.com/Titlehhhh/CaptchaOCR](https://github.com/Titlehhhh/CaptchaOCR).
161
+
162
+ </details>
163
+
164
+ ## License & citation
165
+
166
+ The weights are **Apache-2.0**, as a derivative of `PP-OCRv6_medium_rec`. The training data is
167
+ published under **mixed per-plugin licences** — some rows are non-commercial, so read the dataset
168
+ [LICENSE](https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset/blob/main/LICENSE) before
169
+ commercial use.
170
+
171
+ <details>
172
+ <summary>Version history</summary>
173
+
174
+ * `universal_v1` (2026-10-27) — current weights, 18 762 classes, no weight interpolation (α = 1.0).
175
+ It replaced `all20`, which is no longer downloadable.
176
+ * Interpolation is off on purpose: pulling back towards the base damages the appended rows
177
+ (`LimboFilter/symbols` 0.765 → 0.016). Replay of old data, not a weight soup, is what keeps them.
178
+
179
+ </details>