CaptchaOCR PP-OCRv6: model card (weights pending the training run)

README.md

@@ -0,0 +1,186 @@
+---
+license: apache-2.0
+library_name: transformers
+base_model: PaddlePaddle/PP-OCRv6_medium_rec_safetensors
+tags:
+- captcha
+- ocr
+- minecraft
+- anti-bot
+- pp-ocr
+- image-to-text
+- pytorch
+pipeline_tag: image-to-text
+---
+
+# CaptchaOCR PP-OCRv6 — a text recogniser for Minecraft anti-bot captchas
+
+> ## Status: the weights are **not published yet**. This repository holds the model card only.
+>
+> The fine-tune on the [CaptchaOCR dataset](https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset)
+> has **not been run to completion yet**, so there is **no checkpoint, no `config.json` and no
+> measured metric** in this repository. Nothing here is invented and nothing is extrapolated from a
+> probe run: the numbers quoted below are the **measured numbers of the base model before
+> fine-tuning**, and they are labelled as such.
+>
+> **RU:** репозиторий модели пока **пуст по весам** — здесь только карточка. Обучение на
+> датасете `Titlehhhh/captchaocr-dataset` ещё не завершено, поэтому **метрики появятся после
+> прогона** (плейслолдер ниже — это не выдуманные числа). Выкладка пробного чекпоинта (2 эпохи)
+> сознательно не сделана: он хуже базовой модели.
+
+## What this model is
+
+A fine-tune of **PP-OCRv6 medium** (PaddleOCR's text recogniser) on the CaptchaOCR dataset: the
+captchas that open-source Minecraft anti-bot plugins draw, rendered by this project's own Java
+generators (46 types, 4600 labelled PNG).
+
+* **Architecture:** PP-OCRv6 medium recogniser — PP-LCNetV4 backbone + CTC head,
+  `transformers` class `PPOCRV6SmallRecForTextRecognition`
+  (`model_type: pp_ocrv6_small_rec`), 19.1 M parameters, hidden size 192.
+* **Dictionary:** the base model's own 18 710 classes (index 0 = CTC blank, space at the end).
+* **Input:** the captcha **text band** (the strip of pixels that carries the answer), not the whole
+  frame: resized to height 48, width scaled proportionally and right-padded to 320, BGR channel
+  order, normalised as `(x / 255 - 0.5) / 0.5`.
+* **Output:** per-timestep class probabilities; decoded greedily (CTC: collapse repeats, drop the
+  blank). Greedy decoding is the right decoder here — the answers are random strings, so a language
+  model has nothing to exploit.
+
+**The band is the whole game.** On the 29 dataset types that carry a band, the base model scores
+`acc_micro = 0.5517` on the band and `acc_micro = 0.0345` on the whole frame — about 16× worse on
+the frame. A solver must crop first.
+
+## Provenance and licence
+
+* **Base model:** [`PaddlePaddle/PP-OCRv6_medium_rec_safetensors`](https://huggingface.co/PaddlePaddle/PP-OCRv6_medium_rec_safetensors)
+  — Apache-2.0, by PaddlePaddle.
+* **This model:** that base, fine-tuned on the CaptchaOCR dataset published here:
+  [`Titlehhhh/captchaocr-dataset`](https://huggingface.co/datasets/Titlehhhh/captchaocr-dataset)
+  (46 captcha types · 4600 labelled PNG · 23 upstream plugins).
+* **Licence of the weights:** Apache-2.0, inherited from the base model. The fine-tuning code and
+  this card are released under the same terms.
+* **Licence of the dataset:** *not* Apache-2.0 — the dataset is split by the licence of the
+  upstream plugin of each captcha type. The dataset card has the per-type table.
+
+## Measured results
+
+### This repository's weights
+
+**No metrics yet — the training run has not been completed.** This section will hold exact match
+and CER per type and on the type-held-out split when the fine-tune is done. Until then, treat the
+numbers below as the **starting point**, not as the result of this repository.
+
+### The base model, measured before fine-tuning (baseline)
+
+These are measured numbers of `PaddlePaddle/PP-OCRv6_medium_rec_safetensors` **as it is**, on the
+CaptchaOCR dataset — i.e. the point the fine-tune starts from:
+
+| measurement | value | source |
+|---|---|---|
+| exact match (micro) on the 7 type-held-out types, band crop | 0.5586 | `train/run_ppocr/` zero-step baseline |
+| exact match (median over those 7 types) | 0.60 | `train/run_ppocr/` zero-step baseline |
+| CER (micro) on the same split | 0.2561 | `train/run_ppocr/` zero-step baseline |
+| exact match (micro) on the 29 band-carrying types, band crop | 0.5517 | `reports/bench_models/ppocrv6-medium.json` |
+| exact match (micro) on the same 29 types, whole frame | 0.0345 | `reports/bench_models/ppocrv6-medium.json` |
+
+Both sources are files in the **CapthaModel** project repository; nothing above is a projection.
+
+**Why the weights are not here yet.** A short probe run (2 epochs) of the fine-tune was measured
+on the same held-out types and it came out **worse than the base model** — median exact match
+0.41 after epoch 1 and 0.24 after epoch 3, against 0.60 for the untouched base model. That
+checkpoint is deliberately **not** published: a two-epoch probe is not a model, and shipping it as
+one would misrepresent it.
+
+## How to use it
+
+> The snippet below is the **preprocessing the project itself uses** (`train/ppocr_data.py`), and it
+> was verified end-to-end against the base model on a real sample of the dataset (it decoded the
+> held-out sample `JSH32/Minecaptcha/map/001.png`, band `[25, 77, 79, 35]`, as `PTAY`, which is the
+> gold answer). It is written for the moment the weights are published — until then
+> `from_pretrained("Titlehhhh/captchaocr-ppocrv6")` has nothing to load.
+
+```python
+import json
+
+import numpy as np
+import torch
+from huggingface_hub import hf_hub_download
+from PIL import Image
+from transformers import PPOCRV6SmallRecForTextRecognition
+
+REPO_ID = "Titlehhhh/captchaocr-ppocrv6"
+
+model = PPOCRV6SmallRecForTextRecognition.from_pretrained(REPO_ID).eval()
+# the dictionary comes with the model repository (preprocessor_config.json)
+character_list = json.load(
+    open(hf_hub_download(REPO_ID, "preprocessor_config.json"), encoding="utf-8")
+)["character_list"]
+
+HEIGHT, MAX_WIDTH = 48, 320
+
+
+def prepare(band: Image.Image) -> torch.Tensor:
+    """band -> (1, 3, 48, 320) float tensor."""
+    scale = HEIGHT / max(1, band.height)
+    w = max(1, min(MAX_WIDTH, int(round(band.width * scale))))
+    small = band.convert("RGB").resize((w, HEIGHT), Image.BILINEAR)
+    a = np.asarray(small, dtype=np.float32) / 255.0
+    a = a[:, :, ::-1]            # BGR, as PP-OCRv6 expects
+    a = (a - 0.5) / 0.5         # paddle normalisation
+    canvas = np.zeros((3, HEIGHT, MAX_WIDTH), dtype=np.float32)
+    canvas[:, :, :w] = a.transpose(2, 0, 1)
+    return torch.from_numpy(canvas)[None]
+
+
+def greedy_decode(probs) -> str:
+    """CTC greedy decode: collapse repeats, drop the blank."""
+    best = probs.argmax(dim=-1)[0].tolist()
+    chars, prev = [], -1
+    for k in best:
+        if k != prev and k != 0:
+            chars.append(character_list[k])
+        prev = k
+    return "".join(chars)
+
+
+image = Image.open("captcha.png")
+x, y, w, h = (25, 77, 79, 35)        # the answer band: take it from labels.jsonl
+band = image.crop((x, y, x + w, y + h))
+
+with torch.no_grad():
+    probs = model(pixel_values=prepare(band)).last_hidden_state
+
+print(greedy_decode(probs))
+```
+
+The band (`roi`) is not optional: on scene-like captchas the answer is one line among several, and
+the whole frame makes the model read the wrong line.
+
+## Reproducing
+
+The training code lives in the **CapthaModel** repository:
+
+```powershell
+.venv-gpu\Scripts\python.exe train\finetune_ppocr.py --epochs 30 --batch-size 32
+```
+
+`train/finetune_ppocr.py` fine-tunes `PaddlePaddle/PP-OCRv6_medium_rec_safetensors` on the
+dataset, holds out whole **plugins** (never random splits — a random split leaks the exact gold
+answer of 34 % of the test samples), and reports exact match and CER per type plus the median over
+types. `train/ppocr_data.py` holds the dataset loader and the preprocessing shown above.
+
+## Limitations
+
+* **Only the text band is read.** This is a recogniser, not a solver: it does not reverse strings,
+  does not turn Russian number words into digits and does not compute arithmetic.
+* **No Cyrillic in the dictionary.** The base model's 18 710 classes contain no Cyrillic at all, so
+  the Cyrillic captcha types of the dataset cannot be expressed in this model's alphabet.
+* **The data is synthetic.** Every image is a render of this project's own generators — faithful to
+  the upstream algorithms, but not a screenshot of a live server.
+* **Generalisation is the hard part.** A recogniser can memorise a plugin's style instead of
+  learning the characters; only the type-held-out numbers mean anything.
+
+## Citation
+
+This model is a fine-tune of PP-OCRv6 by PaddlePaddle (Apache-2.0). The dataset is the
+CaptchaOCR dataset, re-rendered from open-source Minecraft anti-bot plugins. See the dataset card
+for the provenance and the per-type licence of every captcha type.