File size: 2,513 Bytes
39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e 39502ae 415711e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | ---
license: apache-2.0
tags:
- layout-detection
- document-ai
- qwen2.5-vl
- lora
- text-with-formula
---
# TeleOCR Layout LoRA (text_with_formula)
This model is a layout-detection LoRA fine-tune of **TeleOCR (Qwen2.5-VL 1.2B)**, merged back into the base weights for direct inference.
> This model is provided for personal amateur study and research only; production availability is not guaranteed.
## Purpose
Built on top of the original TeleOCR layout detection, this model adds a `text_with_formula` label to distinguish:
- `text`: ordinary text blocks
- `text_with_formula`: blocks whose text contains inline formulas (`$...$`)
The output format is identical to native TeleOCR:
```text
<box:xmin ymin xmax ymax><label:text_with_formula><up>
```
Coordinates are relative to a 1036×1036 layout image (integers in 0~1000).
## Training Details
- Base model: TeleOCR (Qwen2.5-VL, ~1.2B)
- Data: OmniDocBench (1651 pages; 1568 train / 83 validation)
- Method: LoRA (r=32, alpha=64, dropout=0.05), training only the language-model projection layer
- Training: 3 epochs, lr=2e-4, batch=1×8 (grad accumulation), bfloat16, cosine schedule
- Data cleaning: strict inline-formula detection (distinguishes currency `$5` from real `$...$`)
## Evaluation (83-page validation set, Hungarian full-category matching + NMS)
| Category | Precision | Recall | F1 |
| --- | --- | --- | --- |
| `text` | 0.767 | 0.723 | 0.744 |
| `text_with_formula` | 0.837 | 0.692 | 0.757 |
| micro overall | 0.779 | 0.713 | 0.744 |
## Usage
```python
from transformers import AutoProcessor, AutoModelForImageTextToText
from TeleOCR.vlm_utils.TeleOCR_client import TeleOCRClient
model_path = "<path to this model>"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, fix_mistral_regex=True)
model = AutoModelForImageTextToText.from_pretrained(
model_path, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
).cuda().eval()
client = TeleOCRClient(backend="transformers", model=model, processor=processor, model_path=model_path)
blocks = client.layout_detect(image) # each block contains type / bbox / angle
```
> Note: this model includes custom code `modeling_naviocr.py`, so `trust_remote_code=True` is required when loading.
## Acknowledgements
Many thanks to the [TeleOCR](https://huggingface.co/XingChen-AGI/TeleOCR) team for open-sourcing the excellent TeleOCR model and code. This model was fine-tuned and studied on top of its base model and toolchain.
|