TrOCR-Sin-Handwritten

A Sinhala text-line recognizer for handwritten text, released with the SinFUND and SinOCR datasets (SriDoc/datasets).

Model

• TrOCR-style vision encoder-decoder (VisionEncoderDecoderModel).
• The encoder was initialized from facebook/deit-base-distilled-patch16-224 (ImageNet weights). The decoder was initialized from NLPC-UOM/SinBERT-large, a RoBERTa model pre-trained on Sinhala text. The cross-attention layers started from random weights.
• No OCR-pretrained TrOCR weights were used.
• Input: one pre-cropped text-line image, resized to 224 x 224.
• Decoding: beam search with 4 beams and a maximum length of 64 tokens.

Training data

This model was trained in two stages:

1. On the SinOCR printed training split. That model is SriDoc/TrOCR-Sin-Printed.
2. Then on the SinOCR handwritten training split (908 lines) for 100 epochs (2,900 steps), with a batch size of 32 and a learning rate of 5e-5.

Results

Character error rate (CER; lower is better) on the SinOCR test splits:

Test split	CER
SinOCR handwritten (227 lines)	0.5253
SinOCR printed (10,000 lines)	0.2246

How the CER is computed:

• Characters are Unicode code points.
• The CER is pooled over the whole test split.

SinOCR has no separate validation split, so the test split was also the evaluation set during training. The lowest test CER occurred at the final checkpoint, so checkpoint selection did not change this result.

Limitations

• Errors remain common. A CER of about 0.53 means roughly one character error for every two characters, so the output needs human checking.
• Test writers were seen in training. The handwritten test split is not writer-disjoint: 72% of its lines come from writers who also appear in training. The CER is therefore probably optimistic for new writers.
• Do not use this model to measure OCR accuracy on the SinFUND test forms. 132 of its training lines are crops of those forms. The dataset repository lists them in sinocr_handwritten_provenance.csv.
• The model reads single, pre-cropped lines. It does not find text on a page.

Usage

from transformers import VisionEncoderDecoderModel, TrOCRProcessor, AutoTokenizer, ViTImageProcessor
from PIL import Image

model = VisionEncoderDecoderModel.from_pretrained("SriDoc/TrOCR-Sin-Handwritten")
tokenizer = AutoTokenizer.from_pretrained("NLPC-UOM/SinBERT-large")
image_processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224")
processor = TrOCRProcessor(image_processor=image_processor, tokenizer=tokenizer)

image = Image.open("line.png").convert("RGB")   # one cropped text line
pixel_values = processor(image, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values)
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

Citation

@article{gunathilaka2025sinfund,
  title   = {SinFUND and SinOCR: Benchmarks for Sinhala Handwritten OCR and Template-Free Form Understanding},
  author  = {Gunathilaka, Kavishka and Hewagama, Danusha and Pushpakumara, Supul and Ambegoda, Thanuja D.},
  journal = {Research Square preprint},
  year    = {2025},
  doi     = {10.21203/rs.3.rs-6976719/v1}
}

Training log

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

The "Cer" and "Validation Loss" columns were measured on the SinOCR handwritten test split.

Training Loss	Epoch	Step	Validation Loss	Cer
0.2915	3.45	100	1.8613	0.6450
0.061	6.9	200	1.8118	0.5707
0.0363	10.34	300	2.3998	0.6420
0.0202	13.79	400	2.4144	0.6353
0.0329	17.24	500	2.4393	0.6577
0.0364	20.69	600	1.9231	0.5679
0.004	24.14	700	2.4344	0.5866
0.0167	27.59	800	3.0998	0.5744
0.0269	31.03	900	2.6785	0.5804
0.0151	34.48	1000	2.2443	0.5916
0.0008	37.93	1100	2.1480	0.5684
0.0067	41.38	1200	2.3553	0.5625
0.0198	44.83	1300	2.1915	0.5492
0.0002	48.28	1400	2.0370	0.5620
0.001	51.72	1500	2.4303	0.6056
0.1666	55.17	1600	2.3324	0.5627
0.0001	58.62	1700	2.8753	0.5582
0.0	62.07	1800	2.5749	0.5355
0.0002	65.52	1900	2.8105	0.5572
0.0	68.97	2000	2.5275	0.5462
0.1231	72.41	2100	2.7452	0.5477
0.0	75.86	2200	2.4278	0.5403
0.0	79.31	2300	3.0099	0.5487
0.0	82.76	2400	3.1290	0.5467
0.0	86.21	2500	2.7705	0.5263
0.0	89.66	2600	2.7828	0.5275
0.0	93.1	2700	3.2488	0.5345
0.0	96.55	2800	3.1309	0.5273
0.0	100.0	2900	2.9839	0.5253

Framework versions

• Transformers 4.35.2
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.1