Add multi-source CRNN recognizer and its character language model (v0.3)

README.md

@@ -11,6 +11,9 @@ tags:
   - pytorch
 datasets:
   - icdar2015
+  - total-text
+  - mlt2019
+  - funsd
 ---
 
 # torchocr weights
@@ -39,7 +42,8 @@ postprocess = DBPostProcessor(**weights.meta["postprocess"])
 | `DBNet_MobileNetV3_Large_05_Weights.PPOCR_V3_EN` | `dbnet_mobilenet_v3_large_05_ppocr_v3_en-9bfd3b59.pth` | 0.60 M | hmean 0.441 ¹ |
 | `DBNet_MobileNetV3_Large_05_Weights.PPOCR_V3_CH` | `dbnet_mobilenet_v3_large_05_ppocr_v3_ch-fc000d1e.pth` | 0.60 M | hmean 0.422 ¹ |
 | `DBNet_ResNet18_VD_Weights.PPOCR_SERVER_V2` | `dbnet_resnet18_vd_ppocr_server_v2-59d99b11.pth` | 12.4 M | hmean 0.414 ¹ |
-| `CRNN_ResNet34_VD_Weights.ICDAR2015` | `crnn_resnet34_vd_ic15-da0782b5.pth` | 27.9 M | **word acc. 0.738** · CER 0.101 ² |
+| `CRNN_ResNet34_VD_Weights.ICDAR2015_TOTALTEXT_MLT19` | `crnn_resnet34_vd_scene-38393899.pth` (+ LM `charlm6_scene-883dd7cd.json`) | 27.9 M | **word acc. 0.791** · Total-Text 0.802 ³ |
+| `CRNN_ResNet34_VD_Weights.ICDAR2015` | `crnn_resnet34_vd_ic15-da0782b5.pth` | 27.9 M | word acc. 0.738 · CER 0.101 ² |
 | `CRNN_ResNet34_VD_Weights.PPOCR_SERVER_V2` | `crnn_resnet34_vd_ppocr_server_v2-803ba4c9.pth` | 27.9 M | word acc. 0.665 ² |
 
 ¹ Line-level detectors converted from PaddleOCR. ICDAR-2015 scores *word* boxes, so adjacent words
@@ -48,18 +52,30 @@ you will see on documents.
 ² 2077 alphanumeric care words of ICDAR-2015 test cropped from the full images with
 `torchocr.ops.crop_quads` on their ground-truth quads, case-insensitive (not the official Task 4.3
 crops). Axis-aligned crops of the same words score 0.579.
+³ Decoded with its declared CTC beam search (width 10) and 6-gram character language model; 0.750 with greedy
+decoding. Same ground-truth-quad protocol; Total-Text curved words use their minimum-area rectangle.
 
-End to end (`OCRPipeline.from_pretrained`, image → text, a word counts when it is detected *and*
-read correctly, case-insensitive, generic vocabulary): **hmean 0.578** with
-`DBNet_ResNet18_VD_Weights.ICDAR2015` + `CRNN_ResNet34_VD_Weights.ICDAR2015`, versus 0.219 with the
-converted PaddleOCR weights (`references/end2end/evaluate.py`).
+End to end (`OCRPipeline.from_pretrained`, image → text; a word counts when it is detected *and*
+read correctly, case-insensitive, generic vocabulary; `references/end2end/evaluate.py`), with
+`DBNet_ResNet18_VD_Weights.ICDAR2015` + `CRNN_ResNet34_VD_Weights.ICDAR2015_TOTALTEXT_MLT19`:
+
+| Test set | Detection hmean | End-to-end hmean | `min_text_score=0.8` |
+| --- | --- | --- | --- |
+| ICDAR-2015 (street scenes) | 0.809 | 0.621 | **0.649** |
+| Total-Text (curved text) | 0.696 | 0.566 | **0.585** |
+| FUNSD (scanned forms) | 0.826 | 0.711 | **0.715** |
+
+`min_text_score=0.8` drops regions read with low confidence; the value maximizes end-to-end hmean on the
+ICDAR-2015 *training* split. The converted PaddleOCR weights reach 0.219 on ICDAR-2015 and 0.17–0.18 detection
+hmean on FUNSD, whose word-level boxes do not match their line-level output.
 
 ```python
 from torchocr import OCRPipeline, load_image
 
 pipeline = OCRPipeline.from_pretrained(
-    "DBNet_ResNet18_VD_Weights.ICDAR2015", "CRNN_ResNet34_VD_Weights.ICDAR2015", device="cuda"
-)
+    "DBNet_ResNet18_VD_Weights.ICDAR2015", "CRNN_ResNet34_VD_Weights.ICDAR2015_TOTALTEXT_MLT19", device="cuda"
+)  # the recognizer's beam search + language model is set up automatically
+pipeline.min_text_score = 0.8
 doc = pipeline(load_image("photo.jpg"))  # doc.text, doc.polygons
 ```
 
@@ -77,7 +93,8 @@ python references/recognition/evaluate.py --root <icdar2015> --weights CRNN_ResN
 
 ### The `ICDAR2015` checkpoints
 
-All three were fine-tuned by torchocr on the ICDAR-2015 training split only; reported weights are the
+All were fine-tuned by torchocr on *training* splits only (ICDAR-2015, plus Total-Text and MLT-2019 for the
+recognizer noted below); reported weights are the
 **last epoch** and no checkpoint or hyper-parameter was selected on the test set.
 
 - `DBNet_ResNet18_VD_Weights.ICDAR2015`: `PPOCR_SERVER_V2` fine-tuned 300 epochs with the recipe below;
@@ -86,6 +103,10 @@ All three were fine-tuned by torchocr on the ICDAR-2015 training split only; rep
   training words, cropped with `crop_quads` from jittered ground-truth quads
   (`references/recognition/train.py`). It keeps PaddleOCR's Chinese + Latin charset but saw only Latin
   scene text, so Chinese accuracy is no longer guaranteed.
+- `CRNN_ResNet34_VD_Weights.ICDAR2015_TOTALTEXT_MLT19`: `PPOCR_SERVER_V2` fine-tuned 10 epochs (bf16) on
+  66k Latin scene words (ICDAR-2015 train ×2, Total-Text train, MLT-2019 Latin crops). Its language model
+  (`charlm6_scene-883dd7cd.json`) is a case-insensitive 6-gram character model built from those *training*
+  transcriptions only; beam width and LM weight were chosen on Total-Text *training* words.
 - `DBNet_MobileNetV3_Large_05_Weights.ICDAR2015`, described next.
 
 #### `DBNet_MobileNetV3_Large_05_Weights.ICDAR2015`

crnn_resnet34_vd_scene-38393899.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38393899588fdde96a8c7c3b18190353c23136170df816739e8806bcf3f59714
+size 111589324