---
license: apache-2.0
tags:
  - layout-detection
  - document-ai
  - qwen2.5-vl
  - lora
  - text-with-formula
---

# TeleOCR Layout LoRA (text_with_formula)

This model is a layout-detection LoRA fine-tune of **TeleOCR (Qwen2.5-VL 1.2B)**, merged back into the base weights for direct inference.

> This model is provided for personal amateur study and research only; production availability is not guaranteed.

## Purpose

Built on top of the original TeleOCR layout detection, this model adds a `text_with_formula` label to distinguish:

- `text`: ordinary text blocks
- `text_with_formula`: blocks whose text contains inline formulas (`$...$`)

The output format is identical to native TeleOCR:

```text
<box:xmin ymin xmax ymax><label:text_with_formula><up>
```

Coordinates are relative to a 1036×1036 layout image (integers in 0~1000).

## Training Details

- Base model: TeleOCR (Qwen2.5-VL, ~1.2B)
- Data: OmniDocBench (1651 pages; 1568 train / 83 validation)
- Method: LoRA (r=32, alpha=64, dropout=0.05), training only the language-model projection layer
- Training: 3 epochs, lr=2e-4, batch=1×8 (grad accumulation), bfloat16, cosine schedule
- Data cleaning: strict inline-formula detection (distinguishes currency `$5` from real `$...$`)

## Evaluation (83-page validation set, Hungarian full-category matching + NMS)

| Category | Precision | Recall | F1 |
| --- | --- | --- | --- |
| `text` | 0.767 | 0.723 | 0.744 |
| `text_with_formula` | 0.837 | 0.692 | 0.757 |
| micro overall | 0.779 | 0.713 | 0.744 |

## Usage

```python
from transformers import AutoProcessor, AutoModelForImageTextToText
from TeleOCR.vlm_utils.TeleOCR_client import TeleOCRClient

model_path = "<path to this model>"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, fix_mistral_regex=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_path, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
).cuda().eval()

client = TeleOCRClient(backend="transformers", model=model, processor=processor, model_path=model_path)
blocks = client.layout_detect(image)  # each block contains type / bbox / angle
```

> Note: this model includes custom code `modeling_naviocr.py`, so `trust_remote_code=True` is required when loading.

## Acknowledgements

Many thanks to the [TeleOCR](https://huggingface.co/XingChen-AGI/TeleOCR) team for open-sourcing the excellent TeleOCR model and code. This model was fine-tuned and studied on top of its base model and toolchain.