This document describes how Artifex turns an image into ranked estimates of region and period and a list of similar reference works. It describes the method; measured results live in the evaluation report of each run (artifacts/evaluation/runs/<run>/report.md) and on the Experiments page.

Research question

To what extent can visual features alone predict the approximate historical period and cultural-geographic region of a painting, and how well do these predictions generalise to artworks, artists and collections not encountered during training? See research_question.md for hypotheses.

Pipeline overview

1. Ingestion — official open-access APIs of The Met and the Cleveland Museum of Art; raw responses are cached verbatim (data/raw/).
2. Normalisation — every record is mapped to a canonical schema (src/artifex/schemas.py). Original strings (dates, cultures, places) are preserved next to normalised values.
3. Labelling — dates become ranges on an astronomical year scale; ranges are assigned to period bins only when they fall mostly inside one bin; region labels come from an explicit, reviewable mapping file.
4. Images — only images the museums release under CC0 are downloaded, validated, resized (max 768 px) and stored locally with thumbnails; exact duplicates are marked.
5. Embeddings — a frozen DINOv2 encoder converts each image to a vector, cached on disk.
6. Splits — leakage-aware grouping and a stratified grouped train/validation/test split.
7. Classifiers — one logistic-regression head for region, one for period, optional heads for narrower traditions; temperature scaling and abstention thresholds fitted on validation data.
8. Retrieval — exact cosine-similarity search over all reference embeddings.
9. Evaluation — baselines, metrics, calibration, ablations and collection-disjoint tests; artefacts read by the website.

Image representation

Encoder. facebook/dinov2-base (ViT-B/14, self-supervised, Apache-2.0), pinned to a specific Hugging Face commit in configs/preprocessing.yaml. The encoder is never fine-tuned. DINOv2 was chosen as a strong, general, image-only representation; it is treated as a baseline to be compared, not as the best possible model.

Embedding. The concatenation of the final-layer CLS token and the mean of the patch tokens (1,536 dimensions), L2-normalised — the representation used for linear evaluation in the DINOv2 paper.

Preprocessing (identical for reference images and uploads, implemented once in src/artifex/preprocess.py):

1. decode with content-based format checks (JPEG, PNG, WebP), decompression-bomb limits;
2. apply EXIF orientation; convert to RGB (transparency composited on white);
3. letterbox: resize so the longer side is 224 px and pad to 224 × 224 with the normalisation mean (which becomes zero after normalisation), so that the whole composition is seen;
4. normalise with ImageNet mean and standard deviation.

Conventional ImageNet preprocessing crops the centre of the image, which can remove compositional information (borders, margins, inscriptions, the format of a handscroll). Centre-cropping is therefore evaluated only as a controlled ablation, alongside greyscale input, a smaller encoder (DINOv2-S) and a non-learned colour/texture descriptor.

Labels

Period

Dates are stored as inclusive ranges [date_start, date_end] in astronomical years (1 CE = 1, 1 BCE = 0, n BCE = 1 − n). Museum APIs encode BCE as negative historical years; these are converted so that differences across the BCE/CE boundary are correct. The original date text is kept (original_date_text), and approximate or uncertain statements are flagged.

The period target is a set of ordered, neutral date bins (e.g. "1450–1549") rather than style names such as "Renaissance", which are specific to European art history and would be meaningless for most of the collection. A work is assigned to a bin only if at least 75 % of its documented range lies inside it; works with wider ranges remain unlabelled for training and are evaluated separately with range-aware metrics. Bins with too few labelled works are not classifier targets. Full definitions: label_taxonomy.md.

Region

Regions describe the cultural-geographic attribution the museum records for the artwork, mapped to historical- geographic zones (e.g. "South Asia", "West & Central Asia"), not modern nation states. Mapping rules are ordered regular expressions in configs/region_mapping.yaml, applied to the museum's culture field first, then place fields, then the artist's recorded nationality. Each record keeps region_basis (which field decided) and region_status: strings naming more than one region are left ambiguous rather than forced into one. Every distinct source string and its mapping is listed in data/manifests/region_mapping_report.csv for review.

Where a museum records an artist's nationality but no culture (common for European paintings), the region label reflects the artist's documented nationality or school, which is not necessarily where the work was made. At the broad-region level this rarely changes the label; at the level of narrower traditions it can.

Classifiers

For each head, a multinomial logistic regression (L-BFGS, balanced class weights, features standardised on the training split) is trained on the frozen embeddings. The regularisation strength C is selected on the validation split by macro-F1. Narrower-tradition heads are trained per broad region when at least two traditions have enough works; at inference a tradition is reported only when the broad-region head did not abstain.

Calibration. Raw classifier scores are not probabilities of historical correctness. A single temperature is fitted on validation logits by minimising negative log-likelihood (temperature scaling). Calibration quality is measured on the test split (expected calibration error, reliability diagrams).

Abstention. For each head, a threshold on the calibrated top score is chosen on the validation split as the lowest threshold whose selective accuracy reaches a target (80 %), subject to a minimum coverage. If the target cannot be met, the threshold enforces the minimum coverage and the interface says so. When a score is below the threshold the analysis reports insufficient evidence. Thresholds are never tuned on the test split.

Out-of-distribution flag. If an upload's nearest training image is less similar than the 1st percentile of validation images' nearest-training similarities, the result is marked as outside the model's well-evaluated scope. This is a heuristic based on distance in the embedding space, not a guarantee of detecting unfamiliar inputs.

Retrieval

The uploaded image's embedding is compared with every cached reference embedding by cosine similarity (an exact matrix product; no approximate index is needed for collections of this size). The top five results are returned, collapsing near-identical images (cosine ≥ 0.97 to an already listed result). Users can explicitly restrict results to the predicted region or period and switch back to the unfiltered ranking. The similarity-weighted labels of the ten nearest works are summarised next to the classifier output, and agreement or disagreement is reported.

A similarity score measures resemblance under the DINOv2 representation. It is not evidence of a shared origin, workshop, or authenticity.

Evidence statements

The analysis page shows short statements such as "Retrieval disagrees: the 10 nearest reference works are mostly documented as …". They are produced by fixed rules in src/artifex/predict.py from the computed scores, thresholds, retrieval labels, training counts and the latest evaluation's per-class recall. No text is generated by a language model, and no statement refers to visual features (brushwork, pigments, motifs) that were not explicitly computed.

Evaluation

Summarised here; the full protocol is in evaluation_protocol.md.

• Grouping and splits. Works are grouped by artist (normalised name, across institutions), by accession lot for anonymous works and manuscript folios/album leaves, and by near-duplicate images (DINOv2 cosine ≥ 0.95). Whole groups are assigned to train/validation/test (70/15/15, stratified by region, fixed seed). At most 25 works per group enter training and evaluation so that a single manuscript cannot dominate a class.
• Baselines. Majority class; logistic regression; linear SVM; similarity-weighted k-NN vote (retrieval only); the mean of logistic-regression and k-NN scores.
• Metrics. Macro-F1, balanced accuracy, per-class precision/recall, confusion matrices; for period additionally within-one-bin accuracy, range-compatible accuracy and mean absolute error in years; calibration error; selective accuracy versus coverage; retrieval region/period precision@k and temporal closeness against a random-ranking baseline.
• Generalisation. Artist-disjoint main split; collection-disjoint experiments (train on one museum, test on the other, removing test works whose artist or group appears in the training museum).
• Ablations. Centre crop vs. whole image, greyscale vs. colour, DINOv2-S vs. DINOv2-B, colour/texture statistics.

Reproducing the experiments

python scripts/ingest_data.py        # fetch metadata (resumable; ~3 h for the Met at its rate limit)
python scripts/build_dataset.py      # normalise, download CC0 images, write the manifest
python scripts/extract_embeddings.py --all-variants
python scripts/train_models.py       # splits + deployed heads
python scripts/evaluate_models.py    # metrics, report, artefacts for the website

Seeds, configuration snapshots, split assignments and model cards are saved with every evaluation run. Museum collections change over time, so re-ingesting later may produce a different dataset version; the manifest and split files of a run identify exactly which records were used.

Limitations

The main scientific, technical and historical limitations — collection bias, Eurocentric and court-centred holdings, cataloguing conventions, label noise, photographic confounds, unsupported traditions, and the meaning of the scores — are discussed in limitations.md. In short: Artifex estimates what is typical of its reference collection, not what is historically true of a given object.