| # RoWeR: RoBERTa Word Error Rate Estimator for OCRed Texts |
|
|
| ```yaml |
| --- |
| language: |
| - en |
| - fr |
| - de |
| - es |
| - it |
| - nl |
| - pt |
| library_name: transformers |
| base_model: FacebookAI/xlm-roberta-base |
| tags: |
| - ocr |
| - quality-estimation |
| - word-error-rate |
| - xlm-roberta |
| - regression |
| - historical-documents |
| - multilingual |
| --- |
| ``` |
| |
| **RoWeR** is a reference-free quality estimation model for OCRed text. It predicts the **Word Error Rate (WER)** of OCR output using **only the OCR text itself**, without requiring a manually corrected reference transcription. |
| |
| RoWeR is based on **XLM-RoBERTa** and was adapted for WER regression using **LoRA**. The models were trained and evaluated on historical OCR data in seven languages: |
| |
| * Portuguese |
| * German |
| * French |
| * Italian |
| * English |
| * Dutch |
| * Spanish |
| |
| The main use case is **OCR quality estimation at scale**: deciding which OCR outputs are good enough for downstream processing, which should be sent to post-OCR correction, and which are too degraded to use reliably. |
| |
| ### Important: output scaling |
| |
| RoWeR was trained to focus on WER values in the **0.0–0.2** range. During training, this interval was linearly scaled to **0–1**, with WER values above `0.2` clipped at the upper boundary. |
| |
| Therefore, the model's raw regression output should be converted back to WER approximately as: |
| |
| ```python |
| estimated_wer = clip(model_output, 0, 1) * 0.2 |
| ``` |
| |
| For example: |
| |
| | Model output | Estimated WER | |
| | -----------: | ---------------------: | |
| | `0.10` | `0.02` — 2% | |
| | `0.25` | `0.05` — 5% | |
| | `0.50` | `0.10` — 10% | |
| | `1.00` | `0.20` — 20% or higher | |
| |
| The model is intended primarily for quality estimation and triage in the **0–20% WER range**. It should not be used to distinguish precisely between highly degraded texts above that range. |
| |
| ## Available models |
| |
| The repository contains both language-specific and multilingual RoWeR models. |
| |
| ### Multilingual models |
| |
| | Folder | Description | |
| | -------------------------------- | ----------------------------------------------------------------------- | |
| | `RoWer-multi-lingual-balanced` | Multilingual model trained with balanced language sampling | |
| | `RoWer-multi-lingual-unbalanced` | Multilingual model trained using the natural training-data distribution | |
| |
| For general use, we recommend starting with a **multilingual model**. In the paper, multilingual training substantially reduced the cross-language calibration problems observed with some language-specific models, while the balanced and unbalanced variants performed similarly overall. |
| |
| ### Language-specific models |
| |
| | Folder | Language | Training corpus | |
| | ------------------------ | ---------- | --------------- | |
| | `RoWeR-RETAS_english` | English | RETAS | |
| | `RoWeR-RETAS_french` | French | RETAS | |
| | `RoWeR-RETAS_german` | German | RETAS | |
| | `RoWeR-RETAS_spanish` | Spanish | RETAS | |
| | `RoWeR-dbnl_dutch` | Dutch | DBNL | |
| | `RoWeR-pocr_english` | English | POCR | |
| | `RoWeR-pocr_french` | French | POCR | |
| | `RoWeR-pocr_german` | German | POCR | |
| | `RoWeR-pocr_italian` | Italian | POCR | |
| | `RoWeR-porto_portuguese` | Portuguese | PORTO | |
| |
| Where multiple models exist for the same language, the variants were trained on different OCR corpora and therefore represent different domains and OCR-noise distributions. |
| |
| ## Repository structure |
| |
| Each model variant contains a LoRA checkpoint and a merged model: |
| |
| ```text |
| <Model folder>/ |
| ├── checkpoint-final/ |
| └── final_merged_model/ |
| ``` |
| |
| For the simplest inference setup, use `final_merged_model`. |
| |
| The merged model folders contain the trained XLM-RoBERTa regression model. The tokenizer should be loaded from the original base model: |
| |
| ```text |
| FacebookAI/xlm-roberta-base |
| ``` |
| |
| ## Installation |
| |
| ```bash |
| pip install torch transformers |
| ``` |
| |
| ## Quick start |
| |
| The following example uses the balanced multilingual model. |
| |
| ```python |
| import torch |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
| REPO_ID = "TomasOsorio/RoweR" |
| MODEL_SUBFOLDER = "RoWer-multi-lingual-balanced/final_merged_model" |
| BASE_MODEL = "FacebookAI/xlm-roberta-base" |
| |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
|
| tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL) |
|
|
| model = AutoModelForSequenceClassification.from_pretrained( |
| REPO_ID, |
| subfolder=MODEL_SUBFOLDER, |
| ) |
| model.to(device) |
| model.eval() |
| |
| ocr_text = """ |
| Th1s is an examp1e of OCRed text with recogn1tion errors. |
| """ |
| |
| inputs = tokenizer( |
| ocr_text, |
| return_tensors="pt", |
| truncation=True, |
| max_length=256, |
| ) |
| inputs = {key: value.to(device) for key, value in inputs.items()} |
| |
| with torch.no_grad(): |
| output = model(**inputs).logits.squeeze().item() |
| |
| # RoWeR was trained with WER [0, 0.2] scaled to [0, 1]. |
| normalized_score = max(0.0, min(1.0, output)) |
| estimated_wer = normalized_score * 0.2 |
| |
| print(f"Estimated WER: {estimated_wer:.4f}") |
| print(f"Estimated word error percentage: {estimated_wer * 100:.2f}%") |
| ``` |
| |
| ## Training data |
| |
| RoWeR was developed using four OCR corpora: |
| |
| * **PORTO** — historical Portuguese OCR data |
| * **DBNL** — historical Dutch texts from the Digital Library of Dutch Literature |
| * **RETAS** — English, French, German, and Spanish OCR data |
| * **POCR** — large-scale English, French, German, and Italian OCR data |
| |
| The training pipeline aligned OCR outputs with reference transcriptions, computed segment-level WER, and applied quality filters to remove unreliable or anomalous samples. |
| |
| ## Model architecture |
| |
| RoWeR uses: |
| |
| * `FacebookAI/xlm-roberta-base` |
| * a sequence-level regression head |
| * LoRA-based parameter-efficient fine-tuning |
| |
| At inference time, the model takes OCR text as input and outputs a single regression value representing estimated OCR degradation. |
| |
| RoWeR does **not** require: |
| |
| * the source document image |
| * OCR-engine confidence scores |
| * a second OCR system |
| * a manually corrected reference transcription |
| |
| ## Intended uses |
| |
| RoWeR is intended for tasks such as: |
| |
| * large-scale OCR quality assessment |
| * document triage before downstream NLP |
| * deciding whether post-OCR correction is likely to be necessary |
| * filtering low-quality OCR text |
| * prioritising documents for manual review |
| * monitoring OCR quality across large digital collections |
| |
| RoWeR estimates OCR quality; it does **not** correct OCR errors. |
| |
| ## Citation |
| |
| If you use RoWeR in your work, please cite: |
| |
| ```bibtex |
| @InProceedings{10.1007/978-3-032-36033-5_1, |
| author="Os{\'o}rio, Tom{\'a}s Freitas and Cardoso, Henrique Lopes", |
| title="RoWeR: RoBERTa Word Error Rate Estimator for OCRed Texts", |
| booktitle="Document Analysis and Recognition -- ICDAR 2026", |
| year="2027", |
| publisher="Springer Nature Switzerland", |
| address="Cham", |
| pages="3--18", |
| } |
| ``` |
| |
