| --- |
| license: apache-2.0 |
| tags: |
| - layout-detection |
| - document-ai |
| - qwen2.5-vl |
| - lora |
| - text-with-formula |
| --- |
| |
| # TeleOCR Layout LoRA (text_with_formula) |
|
|
| This model is a layout-detection LoRA fine-tune of **TeleOCR (Qwen2.5-VL 1.2B)**, merged back into the base weights for direct inference. |
|
|
| > This model is provided for personal amateur study and research only; production availability is not guaranteed. |
|
|
| ## Purpose |
|
|
| Built on top of the original TeleOCR layout detection, this model adds a `text_with_formula` label to distinguish: |
|
|
| - `text`: ordinary text blocks |
| - `text_with_formula`: blocks whose text contains inline formulas (`$...$`) |
|
|
| The output format is identical to native TeleOCR: |
|
|
| ```text |
| <box:xmin ymin xmax ymax><label:text_with_formula><up> |
| ``` |
|
|
| Coordinates are relative to a 1036×1036 layout image (integers in 0~1000). |
|
|
| ## Training Details |
|
|
| - Base model: TeleOCR (Qwen2.5-VL, ~1.2B) |
| - Data: OmniDocBench (1651 pages; 1568 train / 83 validation) |
| - Method: LoRA (r=32, alpha=64, dropout=0.05), training only the language-model projection layer |
| - Training: 3 epochs, lr=2e-4, batch=1×8 (grad accumulation), bfloat16, cosine schedule |
| - Data cleaning: strict inline-formula detection (distinguishes currency `$5` from real `$...$`) |
|
|
| ## Evaluation (83-page validation set, Hungarian full-category matching + NMS) |
|
|
| | Category | Precision | Recall | F1 | |
| | --- | --- | --- | --- | |
| | `text` | 0.767 | 0.723 | 0.744 | |
| | `text_with_formula` | 0.837 | 0.692 | 0.757 | |
| | micro overall | 0.779 | 0.713 | 0.744 | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoProcessor, AutoModelForImageTextToText |
| from TeleOCR.vlm_utils.TeleOCR_client import TeleOCRClient |
| |
| model_path = "<path to this model>" |
| processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, fix_mistral_regex=True) |
| model = AutoModelForImageTextToText.from_pretrained( |
| model_path, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa" |
| ).cuda().eval() |
| |
| client = TeleOCRClient(backend="transformers", model=model, processor=processor, model_path=model_path) |
| blocks = client.layout_detect(image) # each block contains type / bbox / angle |
| ``` |
|
|
| > Note: this model includes custom code `modeling_naviocr.py`, so `trust_remote_code=True` is required when loading. |
| |
| ## Acknowledgements |
| |
| Many thanks to the [TeleOCR](https://huggingface.co/XingChen-AGI/TeleOCR) team for open-sourcing the excellent TeleOCR model and code. This model was fine-tuned and studied on top of its base model and toolchain. |
| |
