ufal/vit-historical-page

Page classification of historical archive scans

Fine-tuned image classifiers that sort scanned archive pages into eleven content categories, so that each page can be sent to the processing it needs: OCR for printed or typed text, handwriting recognition for manuscripts, table extraction for forms, image extraction for drawings and photographs. They look at the page image only, so the language of the page does not matter. The code, the command line and the HTTP service are in atrium-page-classification ^3.

Versions

Every version is a revision (branch) of this repository ^1. They share the eleven categories and differ in their training data and base model: ViT ^2 ^6 ^7, EfficientNetV2 ^8 ^9 ^11 or RegNetY ^12. The current generation is vX.4: five base models, retrained on the licence-compatible public dataset ^10. Name the revision when you load a model: main still holds v4.3, the generation before.

Revision	Base model	Parameters (M)	Input (px)	Top-1, %	Top-3, %	Top-1, unseen documents, %
v1.4	timm/tf_efficientnetv2_m.in21k_ft_in1k	54	384	95.87	99.90	95.60
v4.4	timm/regnety_160.swag_ft_in1k	84	384	95.81	99.79	95.19
v5.4	google/vit-large-patch16-384	305	384	95.58	99.75	95.29
v3.4	google/vit-base-patch16-384	87	384	95.49	99.77	94.96
v2.4	google/vit-base-patch16-224	87	224	94.64	99.81	93.87

Each model was scored on the test partition of its own fold of the dataset (folds 1, 1, 2, 2 and 5; 4,791–4,795 pages each), on 22 October 2026. The five together, averaged, are the code's ensemble (--best, and the default of its HTTP service).

Earlier revisions (2025)

Revision	Base model	Pages	Documents	What changed
v2.0	vit-base-patch16-224	10073	3896	annotations with mistakes, more heterogeneous data
v2.1	vit-base-patch16-224	11940	5002	more diverse pages in each category, fewer annotation mistakes
v2.2	vit-base-patch16-224	15855	5730	same data as v2.1 and some restored pages from v2.0
v3.2	vit-base-patch16-384	15855	5730	same data as v2.2, a larger base model with a higher resolution
v5.2	vit-large-patch16-384	15855	5730	same data as v2.2, the largest base model with a higher resolution
v1.2	efficientnetv2_s.in21k	15855	5730	same data as v2.2, the smallest base model (CNN)
v4.2	efficientnetv2_l.in21k_ft_in1k	15855	5730	same data as v2.2, a CNN base model smaller than the largest
v2.3	vit-base-patch16-224	38625	37328	the new annotation phase, more single-page documents
v3.3	vit-base-patch16-384	38625	37328	same data as v2.3, a larger base model with a higher resolution
v5.3	vit-large-patch16-384	38625	37328	same data as v2.3, the largest base model with a higher resolution
v1.3	efficientnetv2_m.in21k_ft_in1k	38625	37328	same data as v2.3, the smallest base model (CNN)
v4.3	regnety_160.swag_ft_in1k	38625	37328	same data as v2.3, a CNN base model bigger than the smallest (main)

The vX.4 revisions keep only the pages of the CC BY-NC 4.0 dataset: 38,307 pages from 37,316 documents.

How to use

With transformers (4.x) and timm:

import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForImageClassification

CATEGORIES = ["DRAW", "DRAW_L", "LINE_HW", "LINE_P", "LINE_T", "PHOTO", "PHOTO_L", "TEXT", "TEXT_HW", "TEXT_P", "TEXT_T"]

revision = "v4.4"  # name it: `main` is v4.3
processor = AutoImageProcessor.from_pretrained("ufal/vit-historical-page", revision=revision)
model = AutoModelForImageClassification.from_pretrained("ufal/vit-historical-page", revision=revision).eval()

image = Image.open("page.png").convert("RGB")
with torch.no_grad():
    scores = model(**processor(images=image, return_tensors="pt")).logits.softmax(-1)[0]
best = scores.topk(3)
print([(CATEGORIES[i], round(s, 3)) for s, i in zip(best.values.tolist(), best.indices.tolist())])

The model configs name the outputs LABEL_0–LABEL_10; the order is the alphabetical one above.

The code ^3 does the rest: folders and PDFs, the five-model ensemble, tables of the top-N categories, the ATRIUM document record and a paradata log of every run. For example, from a checkout:

python3 run.py --hf -rev v4.4 -f page.png              # one page, one model
python3 run.py --hf -rev v4.4 -f report.pdf --pages 1-3  # PDF pages, rendered at 300 dpi

It is also published as container images (ghcr.io/ufal/atrium-page-classification, and -api for the HTTP service).

Offline, with the weights staged once. On a shared server (a cluster, a Galaxy instance) the weights should be downloaded once, not by every job. --models-dir keeps the fine-tuned models, the base models and the image processors in one folder; with --hf it is filled, and without --hf a run reads only that folder, offline:

for r in v1.4 v2.4 v3.4 v4.4 v5.4; do python3 run.py --hf -rev "$r" --models-dir /srv/atrium-page-models; done
python3 run.py --models-dir /srv/atrium-page-models --best -d pages/   # the five models, averaged

Categories

Label	Pages that hold
DRAW	drawings, maps, paintings, schematics or graphics, possibly with text labels or captions
DRAW_L	drawings and the like laid out as a table, or with a legend formatted as a table
LINE_HW	handwritten text in a tabular or form-like structure
LINE_P	printed text in a tabular or form-like structure
LINE_T	machine-typed text in a tabular or form-like structure
PHOTO	photographs or photographic cutouts, possibly with text captions
PHOTO_L	photographs laid out as a table, or with tabular annotations
TEXT	a mixture of printed, handwritten and typed text, possibly with minor graphical elements
TEXT_HW	only handwritten text, in paragraphs or blocks (not tabular)
TEXT_P	only printed text, in paragraphs or blocks (not tabular)
TEXT_T	only machine-typed text, in paragraphs or blocks (not tabular)

Training data

The pages come from archival documents of 1920–2020 (collections in Prague and Brno), annotated by hand. The dataset, under CC BY-NC 4.0, has 48,499 page images from 37,328 documents. The public record ^10 (LINDAT, with the fold assignments) leaves out 318 pages whose dissemination is restricted; the original annotated dataset with them is ^13. The vX.4 models were trained on the public record's pages only.

Category	Dataset 0	Dataset 1	Dataset 2	Dataset 3
DRAW	1090 (9.1%)	1368 (8.8%)	1472 (9.3%)	2709 (5.6%)
DRAW_L	1091 (9.1%)	1383 (8.9%)	1402 (8.8%)	2921 (6.0%)
LINE_HW	1055 (8.8%)	1113 (7.2%)	1115 (7.0%)	2514 (5.2%)
LINE_P	1092 (9.1%)	1540 (9.9%)	1580 (10.0%)	2439 (5.0%)
LINE_T	1098 (9.2%)	1664 (10.7%)	1668 (10.5%)	9883 (20.4%)
PHOTO	1081 (9.1%)	1632 (10.5%)	1730 (10.9%)	2691 (5.5%)
PHOTO_L	1087 (9.1%)	1087 (7.0%)	1088 (6.9%)	2830 (5.8%)
TEXT	1091 (9.1%)	1587 (10.3%)	1592 (10.0%)	14227 (29.3%)
TEXT_HW	1091 (9.1%)	1092 (7.1%)	1092 (6.9%)	2008 (4.1%)
TEXT_P	1083 (9.1%)	1540 (9.9%)	1633 (10.3%)	2312 (4.8%)
TEXT_T	1081 (9.1%)	1476 (9.5%)	1482 (9.3%)	3965 (8.2%)
Unique PDFs	5001	5694	5729	37328
Total Pages	11,940	15,482	15,854	48,499

The last column (Dataset 3) is the data of the vX.3 models, which used 14,000 of the TEXT pages; the other columns count every page used. Of each set, 80 % was used for training and 10 % each for development and test; the earliest version used 90 % for training and the other 10 % as both development and test set. The categories are unequal because the archive is, not by design.

Training pages: 8,950 (v2.0), 10,745 (v2.1), 14,565 (v2.X), 38,625 (vX.3), 38,307 (vX.4). Test pages: 1,586 (taken from the v2.2 annotations), 4,823 per fold (vX.3), 4,791–4,795 per fold (vX.4: the fold's test partition in the LINDAT record).

Augmentation: during training each of these was applied with a 50 % chance: ColorJitter of brightness, contrast, saturation and hue (0.5 each), a random sharpness change (factor 0.5–1.5), a Gaussian blur (radius 0–2).

Hyperparameters: learning rate 5e-5; batch size 8 (training and evaluation); 3 epochs; warm-up ratio 0.1; evaluation and saving every epoch; the best model by accuracy kept.

Evaluation

The vX.4 figures in the table at the top were measured on 22 October 2026, each model on the test partition of the fold it was trained on. The split is by page: a test page was never trained on, but about a quarter of the test pages come from documents with other pages in training. The last column leaves those out. Two models scored on the same pages do not differ significantly (95 % intervals of about ±0.6 points). The provenance, the figures by group and what was wrong with the figures published here before are in the repository's result/README.md.

Confusion matrices and result tables of the vX.4 models

v1.4, fold 1 (Top-1 95.87 %):

v4.4, fold 1 (Top-1 95.81 %):

v3.4, fold 2 (Top-1 95.49 %):

v5.4, fold 2 (Top-1 95.58 %):

v2.4, fold 5 (Top-1 94.64 %):

• v1.4, fold 1: TOP-1, TOP-3
• v4.4, fold 1: TOP-1, TOP-3
• v3.4, fold 2: TOP-1, TOP-3
• v5.4, fold 2: TOP-1, TOP-3
• v2.4, fold 5: TOP-1, TOP-3
• The five vX.4 models on sample pages, a demonstration of the output: each model's vote BEST_5_models_TOP-1.csv and the average BEST_5_models_AVG_TOP-3.csv

Base-model sweep of 2025 (vX.3), not held-out

These figures were measured on a 5,449-page evaluation set, most of whose pages are training pages (its selection script missed zero-padded page numbers and ids with a letter). They chose each base model's fold, and they are not held-out accuracy. The stored tables do not all reproduce them either: v4.3 gives 98.92 % there, not 99.16 %.

Base model	Revision	max_cat	Best_Prec (%)	Best_Acc (%)	Fold
google/vit-base-patch16-224	v2.3	14,000	98.79	98.79	5
google/vit-base-patch16-384	v3.3	14,000	98.92	98.92	2
google/vit-large-patch16-384	v5.3	14,000	99.12	99.12	2
microsoft/dit-base-finetuned-rvlcdip	v9.3	14,000	98.71	98.72	3
microsoft/dit-large-finetuned-rvlcdip	v10.3	14,000	98.66	98.66	3
microsoft/dit-large	v11.3	14,000	98.53	98.53	2
timm/regnety_120.sw_in12k_ft_in1k	v12.3	14,000	98.29	98.29	3
timm/regnety_160.swag_ft_in1k	v4.3	14,000	99.17	99.16	1
timm/regnety_640.see	v6.3	14,000	98.79	98.79	5
timm/tf_efficientnetv2_l.in21k_ft_in1k	v8.3	14,000	98.62	98.62	5
timm/tf_efficientnetv2_m.in21k_ft_in1k	v1.3	14,000	98.83	98.83	1
timm/tf_efficientnetv2_s.in21k	v7.3	14,000	97.90	97.87	1

Earlier vX.2 models (2025-07)

Scored on the v2.2 evaluation set (1,583–1,586 pages), whose overlap with their training data was never checked: read the figures as examples, not as held-out accuracy.

Revision	Top-1, %	Top-3, %
v1.2	97.73	99.87
v2.2	97.54	99.94
v3.2	97.41	99.94
v4.2	97.86	99.87
v5.2	97.73	99.87

The result tables are listed in the repository's README ^3.

Limits

• One collection. The models were trained and tested on Czech archaeological archives of 1920–2020. On another collection, check a sample by hand first.
• Page-level, single-fold results. Each model was tested on one fold, split by page; the unseen-documents column is the closest measure of documents the model has not seen. It is not five-fold cross-validation.
• Neighbouring categories are the hard cases, TEXT and TEXT_T above all. A page that mixes several kinds of content gets the category of what dominates.
• Scores are not calibrated confidence. They are the model's probabilities; the code renormalises them over the N categories it keeps. A low top score with a close second marks a page worth a look, and so does a page on which the five models disagree.

Citation

@article{lutsai2026pageclassifier,
  author  = {Lutsai, Kateryna and K\v{r}iv\'ankov\'a, Dana and Stra\v{n}\'ak, Pavel and Nov\'ak, David},
  title   = {Page image classifier fine-tuned on century-spanning archives of scanned documents for further content-specific processing},
  journal = {arXiv preprint arXiv:2606.07558},
  year    = {2026},
  doi     = {10.48550/arXiv.2606.07558}
}

@article{lutsai2025pageclassification,
  author  = {Lutsai, Kateryna and Stra\v{n}\'ak, Pavel},
  title   = {Page image classification for content-specific data processing},
  journal = {arXiv preprint arXiv:2507.21114},
  year    = {2025},
  doi     = {10.48550/arXiv.2507.21114}
}

@misc{lutsai2026dataset,
  author       = {Lutsai, Kateryna and K\v{r}iv\'ankov\'a, Dana},
  title        = {Annotated page images from the (archaeological) historical archive},
  publisher    = {LINDAT/CLARIAH-CZ},
  version      = {1.1.0},
  year         = {2026},
  url          = {http://hdl.handle.net/20.500.12800/1-6184}
}

The first paper compares the vision-only models with CLIP on 649,508 unlabelled archive pages; the second covers the data, the benchmark of image-classification approaches (Random Forest, EfficientNetV2, RegNetY, DiT, ViT and CLIP) and the system. The code has its own CITATION.cff ^3.

Acknowledgements

Developed by UFAL ^5 and funded by ATRIUM ^4 (Horizon Europe, grant agreement 101132163); shared by ATRIUM and UFAL. Contact: lutsai.k@gmail.com.

© 2022 UFAL & ATRIUM