openocrv2 / README.md
Yesianrohn's picture
Upload 12 files
a93652f verified
|
Raw History Blame Contribute Delete
8.51 kB
---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- ocr
- text-detection
- text-recognition
- openocrv2
---
# OpenOCRv2
Text detection and recognition for scene/document images, deployable with plain
`torch` + `transformers`. Three models live in one repository, in three
subfolders β€” download only what you need with `subfolder=`.
| Subfolder | Model | Architecture | Params | Weights |
| --- | --- | --- | ---: | ---: |
| `det/` | **OpenOCRv2_Det** | HGNetv2-b4 + DCNv2 + polygon regression head | 9.36 M | 36 MB |
| `rec_ctc/` | **OpenOCRv2_Rec_CTC** | ConvNeXt + 2D-RoPE text encoder + linear CTC head | 21.97 M | 85 MB |
| `rec_ar/` | **OpenOCRv2_Rec_AR** | Same encoder + autoregressive decoder (RoPE, KV cache) | 36.66 M | 141 MB |
## Requirements
For Windows CPU inference, install these versions in a fresh Python 3.11
virtual environment:
```powershell
pip install torch==2.14.0 transformers==5.22.2 safetensors==0.8.0 numpy==2.2.6 pillow==12.3.0 opencv-python-headless==5.0.0.93
```
For NVIDIA GPU inference, install a CUDA-enabled build of `torch` using the
[PyTorch installation guide](https://pytorch.org/get-started/locally/). Use the
same versions of the other packages.
`opencv-python-headless` is only needed for the crop step (`det.ocr(...)`).
Polygon detection (`det.detect(...)`) and recognition (`rec.recognize(...)` on
already-cropped images) work without it.
> If you see `Run pip install cv2` β€” that is transformers' fallback hint and the
> package name does not exist. Install `opencv-python-headless` instead.
OpenOCRv2 does not require `torchvision`. If importing the model raises
`operator torchvision::nms does not exist`, use a clean environment with the
packages above, or install a `torchvision` build that matches your `torch`.
## Usage
- **CTC (`rec_ctc`)** is the faster recognizer. Use it when throughput or latency matters most.
- **AR (`rec_ar`)** is the accuracy-focused recognizer. It decodes one token at a time, so it is slower.
Each example runs on CUDA when `torch.cuda.is_available()` is true and uses CPU
otherwise. Pass NumPy images as shown. The models move their input tensors to the
selected device, while OpenCV cropping stays on the CPU.
`det.detect()` and `rec.recognize()` already disable gradient recording for the
model forward passes. `det.ocr()` uses both methods.
### Full-page OCR
Replace `page.jpg` with a page or scene image. Each example prints the text in
reading order and each region's recognition score, text, and quadrilateral.
#### CTC (speed-focused)
```python
import numpy as np
import torch
from PIL import Image
from transformers import AutoModel
REPO = "Yesianrohn/openocrv2"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
with Image.open("page.jpg") as source:
image = np.asarray(source.convert("RGB"))
det = AutoModel.from_pretrained(REPO, subfolder="det", trust_remote_code=True).to(device).eval()
rec = AutoModel.from_pretrained(REPO, subfolder="rec_ctc", trust_remote_code=True).to(device).eval()
result = det.ocr(image, rec)
print("CTC β€” text in reading order")
print("\n".join(result["text"]))
print("Score, text, quadrilateral")
for region in result["regions"]:
print(f'{region["rec_score"]:.3f}', region["text"], region["quad"].tolist())
```
#### AR (accuracy-focused)
```python
import numpy as np
import torch
from PIL import Image
from transformers import AutoModel
REPO = "Yesianrohn/openocrv2"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
with Image.open("page.jpg") as source:
image = np.asarray(source.convert("RGB"))
det = AutoModel.from_pretrained(REPO, subfolder="det", trust_remote_code=True).to(device).eval()
rec = AutoModel.from_pretrained(REPO, subfolder="rec_ar", trust_remote_code=True).to(device).eval()
result = det.ocr(image, rec)
print("AR β€” text in reading order")
print("\n".join(result["text"]))
print("Score, text, quadrilateral")
for region in result["regions"]:
print(f'{region["rec_score"]:.3f}', region["text"], region["quad"].tolist())
```
### Detection only
Pass the full RGB image to the detector. It returns polygons in original image
pixel coordinates and a detection score for each polygon.
```python
import numpy as np
import torch
from PIL import Image
from transformers import AutoModel
REPO = "Yesianrohn/openocrv2"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
with Image.open("page.jpg") as source:
image = np.asarray(source.convert("RGB"))
det = AutoModel.from_pretrained(REPO, subfolder="det", trust_remote_code=True).to(device).eval()
page = det.detect([image])[0]
for polygon, score in zip(page["polygons"], page["scores"]):
print(f"{score:.3f}", polygon.tolist())
```
### Recognition only
Use one upright text line per crop. The recognizers take a list of BGR
`uint8` arrays and return `(text, confidence)` pairs in input order.
#### CTC (speed-focused)
```python
import numpy as np
import torch
from PIL import Image
from transformers import AutoModel
REPO = "Yesianrohn/openocrv2"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
crop_paths = ["line_1.jpg", "line_2.jpg"]
crops = []
for path in crop_paths:
with Image.open(path) as source:
rgb = np.asarray(source.convert("RGB"))
crops.append(rgb[..., ::-1].copy()) # RGB -> BGR
rec = AutoModel.from_pretrained(REPO, subfolder="rec_ctc", trust_remote_code=True).to(device).eval()
print("CTC β€” text and confidence")
for path, (text, confidence) in zip(crop_paths, rec.recognize(crops)):
print(f"{path}: {text} ({confidence:.3f})")
```
#### AR (accuracy-focused)
```python
import numpy as np
import torch
from PIL import Image
from transformers import AutoModel
REPO = "Yesianrohn/openocrv2"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
crop_paths = ["line_1.jpg", "line_2.jpg"]
crops = []
for path in crop_paths:
with Image.open(path) as source:
rgb = np.asarray(source.convert("RGB"))
crops.append(rgb[..., ::-1].copy()) # RGB -> BGR
rec = AutoModel.from_pretrained(REPO, subfolder="rec_ar", trust_remote_code=True).to(device).eval()
print("AR β€” text and confidence")
for path, (text, confidence) in zip(crop_paths, rec.recognize(crops)):
print(f"{path}: {text} ({confidence:.3f})")
```
If you crop yourself, keep these two details β€” they are the difference between
correct output and silently degraded output:
- Quad order is TL, TR, BR, BL after `cv2.minAreaRect` + sort-by-x, not
sort-by-angle.
- Crops taller than 1.5Γ— their width must be rotated 90Β° before recognition,
or vertical text comes back wrong.
`det.ocr` already does both. `modeling_openocrv2_det` exposes the helpers as
`polygon_to_quad` and `get_rotate_crop_image` if you want to reuse them.
### Batching
`rec.recognize(crops, batch_size=32, micro_batch_size=8)`:
- `batch_size` is the **logical** batch β€” it decides the padding size, and the
attention length changes with it, so predictions depend on it.
- `micro_batch_size` is how many run per forward pass (memory only, does not
affect output).
Accuracy figures were produced with `batch_size=32`. Changing it will shift
results slightly.
## Preprocessing contracts
These must match the pipeline the weights were frozen with, or output changes:
- **Detection:** RGB, letterbox to 1600Γ—1600 with zero padding, ImageNet
normalisation (mean `[123.675, 116.28, 103.53]`, std `[58.395, 57.12, 57.375]`).
- **Recognition:** aspect-preserving resize onto the ladder
`[[64,64], [96,48], [112,40], [128,32]]` (width, height), then
`(x/255 - 0.5) / 0.5`, zero-padded to the batch maximum.
Defaults live in each `config.json` (`preprocess`, `image_size`, thresholds) and
are applied automatically.
## Files
```text
modeling_openocrv2_det.py
modeling_openocrv2_rec_ctc.py
modeling_openocrv2_rec_ar.py
det/
β”œβ”€β”€ config.json
└── model.safetensors # 447 tensors, incl. the 14x28 polygon basis
rec_ctc/ rec_ar/
β”œβ”€β”€ config.json
β”œβ”€β”€ model.safetensors
└── characters.json # 19685 / 19687-entry alphabet, bundled
```
The `auto_map` entries resolve modeling modules from the repository root.
The `subfolder` argument selects the config and weights. To load a local
download, pass the path to its root in place of `REPO` and keep `subfolder=`.
The character table ships inside each recogniser repo, so decoding needs nothing
beyond `from_pretrained`.