maclenn77/ticha-trocr-base

TrOCR base for Colonial Zapotec and Spanish manuscripts (Ticha)

Handwritten text recognition for line images of 17th to 19th century manuscripts from Oaxaca, Mexico, written in Colonial Zapotec and Spanish. It is microsoft/trocr-base-handwritten fine-tuned on maclenn77/ticha-htr-lines, built from the transcriptions of Ticha: a digital text explorer for Colonial Zapotec (Haverford College).

The repository holds the two models of the best system:

Folder	Model	Role
/ (root)	Left-to-right TrOCR (run E)	Reads the line; returns the 10 best candidates with beam search
r2l/	Right-to-left TrOCR	Rescores each candidate read backwards; the best combined score wins

Code, training notebooks and the evaluation: github.com/Maclenn77/ticha (notebooks/train/, notebooks/ticha_report_metrics.ipynb).

Results

Character and word error rates in %, lower is better. Test is the 936 test lines of dataset v1.0 without Za687, a test document whose pages are also in train (Th687); validation has 710 lines.

System	Validation CER	Test CER	Test WER	Spanish	Spanish with some Zapotec	Zapotec
Pretrained trocr-base-handwritten, no fine-tuning (2 beams)	71.8	68.8	103.7	65.3	74.2	71.6
This model alone, 2 beams	16.2	14.4	41.8	10.6	21.1	16.9
This model alone, 10 beams	15.7	14.1	41.0	10.5	20.5	16.5
10 beams + right-to-left rerank (w = 0.3)	15.3	13.6	40.2	10.1	19.9	15.8

The rerank gain over 10 beams alone is 0.55 points of test CER (95% interval 0.28 to 0.80, paired bootstrap over whole documents). On a degraded copy of each test line (blur, noise, faded ink, stains) the best system's CER is 13.7%.

Transcription conventions

The model writes a diplomatic transcription: what is on the page. In the training text, abbreviation expansions in brackets were deleted (p[es]os becomes pos), glosses such as [=Juan] were deleted, uncertain readings such as (Santigo?) were kept as Santigo, and lines with damage notes or illegible text ([cortado], [roto], [...], (?), #) were dropped. It does not expand abbreviations.

Training

Both models start from microsoft/trocr-base-handwritten and train on dataset v1.0 (revision b9747c9): 5,305 cleaned original train lines plus 21,220 augmented copies (26,525 lines), with label smoothing 0.1, learning rate 5e-5, batch size 8, 12 epochs, bf16 on one NVIDIA L4, and the epoch with the best validation CER kept.

• Left-to-right (root): linear learning-rate schedule. This was the best of an ablation over label smoothing, the learning-rate schedule and augmentation.
• Right-to-left (r2l/): the same, with a cosine schedule and 5% warmup, trained to write each line's tokens in reverse order. Alone it reads validation at 18.5% CER; it is meant only for reranking.

Both models expect the decoder start token 0 (<s>), which is set in their configs here.

Usage

Left-to-right model alone:

import torch
from PIL import Image
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

processor = TrOCRProcessor.from_pretrained("maclenn77/ticha-trocr-base")
model = VisionEncoderDecoderModel.from_pretrained("maclenn77/ticha-trocr-base").eval()

image = Image.open("line.png").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values
with torch.inference_mode():
    ids = model.generate(pixel_values, num_beams=10, max_length=64, early_stopping=True)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Best system, 10 beams reranked with the right-to-left model:

from transformers.models.vision_encoder_decoder.modeling_vision_encoder_decoder import shift_tokens_right

r2l = VisionEncoderDecoderModel.from_pretrained("maclenn77/ticha-trocr-base", subfolder="r2l").eval()
tok = processor.tokenizer
START, BOS, PAD, EOS = 0, tok.cls_token_id, tok.pad_token_id, tok.sep_token_id
BEAMS, W = 10, 0.3  # W chosen on validation


def inner_tokens(ids):
    ids = [int(i) for i in ids]
    while ids and ids[0] in (START, BOS):
        ids = ids[1:]
    return ids[:ids.index(EOS)] if EOS in ids else [i for i in ids if i != PAD]


@torch.inference_mode()
def log_probs(m, pixel_values, label_lists):
    width = max(map(len, label_lists))
    labels = torch.full((len(label_lists), width), -100, dtype=torch.long)
    for row, ids in enumerate(label_lists):
        labels[row, :len(ids)] = torch.tensor(ids)
    logits = m(pixel_values=pixel_values, decoder_input_ids=shift_tokens_right(labels, PAD, START)).logits
    logp = logits.float().log_softmax(-1).gather(-1, labels.clamp(min=0).unsqueeze(-1)).squeeze(-1)
    return (logp * (labels != -100)).sum(-1).tolist()


with torch.inference_mode():
    seqs = model.generate(pixel_values, num_beams=BEAMS, num_return_sequences=BEAMS, max_length=64, early_stopping=True)
inners = [inner_tokens(s) for s in seqs.tolist()]
repeated = pixel_values.repeat_interleave(BEAMS, 0)
l2r_scores = log_probs(model, repeated, [[BOS] + i + [EOS] for i in inners])
r2l_scores = log_probs(r2l, repeated, [[BOS] + i[::-1] + [EOS] for i in inners])
best = max(range(BEAMS), key=lambda k: ((1 - W) * l2r_scores[k] + W * r2l_scores[k]) / (len(inners[k]) + 2))
print(tok.decode(inners[best], skip_special_tokens=True).strip())

Limitations

• Trained on about 5,300 real lines from 93 documents; new hands and document types may read worse. The model memorizes its training lines (0.7% CER on them against 16.2% on validation).
• Zapotec lines (15.8% CER) and lines mixing Spanish and Zapotec (19.9%, all from one test document) are read worse than Spanish lines (10.1%).
• Input is a single segmented text line; page segmentation is not included.
• The test score leaves out damaged and uncertain lines (see Transcription conventions), so it describes legible lines.

License and citation

CC BY-NC-SA 4.0, the license of the training data. The page images and transcriptions come from Ticha (Haverford College); please credit Ticha and cite the dataset (DOI 10.57967/hf/11240) when using this model.