README.md

# RoWeR: RoBERTa Word Error Rate Estimator for OCRed Texts

```yaml
---
language:
- en
- fr
- de
- es
- it
- nl
- pt
library_name: transformers
base_model: FacebookAI/xlm-roberta-base
tags:
- ocr
- quality-estimation
- word-error-rate
- xlm-roberta
- regression
- historical-documents
- multilingual
---
```

**RoWeR** is a reference-free quality estimation model for OCRed text. It predicts the **Word Error Rate (WER)** of OCR output using **only the OCR text itself**, without requiring a manually corrected reference transcription.

RoWeR is based on **XLM-RoBERTa** and was adapted for WER regression using **LoRA**. The models were trained and evaluated on historical OCR data in seven languages:

* Portuguese
* German
* French
* Italian
* English
* Dutch
* Spanish

The main use case is **OCR quality estimation at scale**: deciding which OCR outputs are good enough for downstream processing, which should be sent to post-OCR correction, and which are too degraded to use reliably.

### Important: output scaling

RoWeR was trained to focus on WER values in the **0.0–0.2** range. During training, this interval was linearly scaled to **0–1**, with WER values above `0.2` clipped at the upper boundary.

Therefore, the model's raw regression output should be converted back to WER approximately as:

```python
estimated_wer = clip(model_output, 0, 1) * 0.2
```

For example:

| Model output |          Estimated WER |
| -----------: | ---------------------: |
|       `0.10` |            `0.02` — 2% |
|       `0.25` |            `0.05` — 5% |
|       `0.50` |           `0.10` — 10% |
|       `1.00` | `0.20` — 20% or higher |

The model is intended primarily for quality estimation and triage in the **0–20% WER range**. It should not be used to distinguish precisely between highly degraded texts above that range.

## Available models

The repository contains both language-specific and multilingual RoWeR models.

### Multilingual models

| Folder                           | Description                                                             |
| -------------------------------- | ----------------------------------------------------------------------- |
| `RoWer-multi-lingual-balanced`   | Multilingual model trained with balanced language sampling              |
| `RoWer-multi-lingual-unbalanced` | Multilingual model trained using the natural training-data distribution |

For general use, we recommend starting with a **multilingual model**. In the paper, multilingual training substantially reduced the cross-language calibration problems observed with some language-specific models, while the balanced and unbalanced variants performed similarly overall.

### Language-specific models

| Folder                   | Language   | Training corpus |
| ------------------------ | ---------- | --------------- |
| `RoWeR-RETAS_english`    | English    | RETAS           |
| `RoWeR-RETAS_french`     | French     | RETAS           |
| `RoWeR-RETAS_german`     | German     | RETAS           |
| `RoWeR-RETAS_spanish`    | Spanish    | RETAS           |
| `RoWeR-dbnl_dutch`       | Dutch      | DBNL            |
| `RoWeR-pocr_english`     | English    | POCR            |
| `RoWeR-pocr_french`      | French     | POCR            |
| `RoWeR-pocr_german`      | German     | POCR            |
| `RoWeR-pocr_italian`     | Italian    | POCR            |
| `RoWeR-porto_portuguese` | Portuguese | PORTO           |

Where multiple models exist for the same language, the variants were trained on different OCR corpora and therefore represent different domains and OCR-noise distributions.

## Repository structure

Each model variant contains a LoRA checkpoint and a merged model:

```text
<Model folder>/
├── checkpoint-final/
└── final_merged_model/
```

For the simplest inference setup, use `final_merged_model`.

The merged model folders contain the trained XLM-RoBERTa regression model. The tokenizer should be loaded from the original base model:

```text
FacebookAI/xlm-roberta-base
```

## Installation

```bash
pip install torch transformers
```

## Quick start

The following example uses the balanced multilingual model.

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO_ID = "TomasOsorio/RoweR"
MODEL_SUBFOLDER = "RoWer-multi-lingual-balanced/final_merged_model"
BASE_MODEL = "FacebookAI/xlm-roberta-base"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

model = AutoModelForSequenceClassification.from_pretrained(
    REPO_ID,
    subfolder=MODEL_SUBFOLDER,
)
model.to(device)
model.eval()

ocr_text = """
Th1s is an examp1e of OCRed text with recogn1tion errors.
"""

inputs = tokenizer(
    ocr_text,
    return_tensors="pt",
    truncation=True,
    max_length=256,
)
inputs = {key: value.to(device) for key, value in inputs.items()}

with torch.no_grad():
    output = model(**inputs).logits.squeeze().item()

# RoWeR was trained with WER [0, 0.2] scaled to [0, 1].
normalized_score = max(0.0, min(1.0, output))
estimated_wer = normalized_score * 0.2

print(f"Estimated WER: {estimated_wer:.4f}")
print(f"Estimated word error percentage: {estimated_wer * 100:.2f}%")
```

## Training data

RoWeR was developed using four OCR corpora:

* **PORTO** — historical Portuguese OCR data
* **DBNL** — historical Dutch texts from the Digital Library of Dutch Literature
* **RETAS** — English, French, German, and Spanish OCR data
* **POCR** — large-scale English, French, German, and Italian OCR data

The training pipeline aligned OCR outputs with reference transcriptions, computed segment-level WER, and applied quality filters to remove unreliable or anomalous samples.

## Model architecture

RoWeR uses:

* `FacebookAI/xlm-roberta-base`
* a sequence-level regression head
* LoRA-based parameter-efficient fine-tuning

At inference time, the model takes OCR text as input and outputs a single regression value representing estimated OCR degradation.

RoWeR does **not** require:

* the source document image
* OCR-engine confidence scores
* a second OCR system
* a manually corrected reference transcription

## Intended uses

RoWeR is intended for tasks such as:

* large-scale OCR quality assessment
* document triage before downstream NLP
* deciding whether post-OCR correction is likely to be necessary
* filtering low-quality OCR text
* prioritising documents for manual review
* monitoring OCR quality across large digital collections

RoWeR estimates OCR quality; it does **not** correct OCR errors.

## Citation

If you use RoWeR in your work, please cite:

```bibtex
@InProceedings{10.1007/978-3-032-36033-5_1,
author="Os{\'o}rio, Tom{\'a}s Freitas and Cardoso, Henrique Lopes",
title="RoWeR: RoBERTa Word Error Rate Estimator for OCRed Texts",
booktitle="Document Analysis and Recognition -- ICDAR 2026",
year="2027",
publisher="Springer Nature Switzerland",
address="Cham",
pages="3--18",
}
```