| <p>This document describes how Artifex turns an image into ranked estimates of region and period and a list of similar | |
| reference works. It describes the <em>method</em>; measured results live in the evaluation report of each run | |
| (<code>artifacts/evaluation/runs/<run>/report.md</code>) and on the Experiments page.</p> | |
| <h2>Research question</h2> | |
| <p><em>To what extent can visual features alone predict the approximate historical period and cultural-geographic region of | |
| a painting, and how well do these predictions generalise to artworks, artists and collections not encountered during | |
| training?</em> See <a href="#/docs/research_question">research_question.md</a> for hypotheses.</p> | |
| <h2>Pipeline overview</h2> | |
| <ol> | |
| <li><strong>Ingestion</strong> — official open-access APIs of The Met and the Cleveland Museum of Art; raw responses are cached | |
| verbatim (<code>data/raw/</code>).</li> | |
| <li><strong>Normalisation</strong> — every record is mapped to a canonical schema (<code>src/artifex/schemas.py</code>). Original strings | |
| (dates, cultures, places) are preserved next to normalised values.</li> | |
| <li><strong>Labelling</strong> — dates become ranges on an astronomical year scale; ranges are assigned to period bins only when | |
| they fall mostly inside one bin; region labels come from an explicit, reviewable mapping file.</li> | |
| <li><strong>Images</strong> — only images the museums release under CC0 are downloaded, validated, resized (max 768 px) and stored | |
| locally with thumbnails; exact duplicates are marked.</li> | |
| <li><strong>Embeddings</strong> — a frozen DINOv2 encoder converts each image to a vector, cached on disk.</li> | |
| <li><strong>Splits</strong> — leakage-aware grouping and a stratified grouped train/validation/test split.</li> | |
| <li><strong>Classifiers</strong> — one logistic-regression head for region, one for period, optional heads for narrower traditions; | |
| temperature scaling and abstention thresholds fitted on validation data.</li> | |
| <li><strong>Retrieval</strong> — exact cosine-similarity search over all reference embeddings.</li> | |
| <li><strong>Evaluation</strong> — baselines, metrics, calibration, ablations and collection-disjoint tests; artefacts read by the website.</li> | |
| </ol> | |
| <h2>Image representation</h2> | |
| <p><strong>Encoder.</strong> <code>facebook/dinov2-base</code> (ViT-B/14, self-supervised, Apache-2.0), pinned to a specific Hugging Face | |
| commit in <code>configs/preprocessing.yaml</code>. The encoder is never fine-tuned. DINOv2 was chosen as a strong, general, | |
| <em>image-only</em> representation; it is treated as a baseline to be compared, not as the best possible model.</p> | |
| <p><strong>Embedding.</strong> The concatenation of the final-layer CLS token and the mean of the patch tokens (1,536 dimensions), | |
| L2-normalised — the representation used for linear evaluation in the DINOv2 paper.</p> | |
| <p><strong>Preprocessing</strong> (identical for reference images and uploads, implemented once in <code>src/artifex/preprocess.py</code>):</p> | |
| <ol> | |
| <li>decode with content-based format checks (JPEG, PNG, WebP), decompression-bomb limits;</li> | |
| <li>apply EXIF orientation; convert to RGB (transparency composited on white);</li> | |
| <li><strong>letterbox</strong>: resize so the longer side is 224 px and pad to 224 × 224 with the normalisation mean (which becomes | |
| zero after normalisation), so that the <em>whole composition</em> is seen;</li> | |
| <li>normalise with ImageNet mean and standard deviation.</li> | |
| </ol> | |
| <p>Conventional ImageNet preprocessing crops the centre of the image, which can remove compositional information | |
| (borders, margins, inscriptions, the format of a handscroll). Centre-cropping is therefore evaluated only as a | |
| controlled ablation, alongside greyscale input, a smaller encoder (DINOv2-S) and a non-learned colour/texture | |
| descriptor.</p> | |
| <h2>Labels</h2> | |
| <h3>Period</h3> | |
| <p>Dates are stored as inclusive ranges <code>[date_start, date_end]</code> in <strong>astronomical years</strong> (1 CE = 1, 1 BCE = 0, | |
| n BCE = 1 − n). Museum APIs encode BCE as negative historical years; these are converted so that differences across | |
| the BCE/CE boundary are correct. The original date text is kept (<code>original_date_text</code>), and approximate or uncertain | |
| statements are flagged.</p> | |
| <p>The period target is a set of ordered, <strong>neutral date bins</strong> (e.g. "1450–1549") rather than style names such as | |
| "Renaissance", which are specific to European art history and would be meaningless for most of the collection. | |
| A work is assigned to a bin only if at least 75 % of its documented range lies inside it; works with wider ranges | |
| remain unlabelled for training and are evaluated separately with range-aware metrics. Bins with too few labelled works | |
| are not classifier targets. Full definitions: <a href="#/docs/label_taxonomy">label_taxonomy.md</a>.</p> | |
| <h3>Region</h3> | |
| <p>Regions describe the cultural-geographic attribution the museum records <strong>for the artwork</strong>, mapped to historical- | |
| geographic zones (e.g. "South Asia", "West & Central Asia"), not modern nation states. Mapping rules are ordered regular | |
| expressions in <code>configs/region_mapping.yaml</code>, applied to the museum's culture field first, then place fields, then the | |
| artist's recorded nationality. Each record keeps <code>region_basis</code> (which field decided) and <code>region_status</code>: | |
| strings naming more than one region are left <strong>ambiguous</strong> rather than forced into one. Every distinct source string and | |
| its mapping is listed in <code>data/manifests/region_mapping_report.csv</code> for review.</p> | |
| <p>Where a museum records an artist's nationality but no culture (common for European paintings), the region label reflects | |
| the artist's documented nationality or school, which is not necessarily where the work was made. At the broad-region | |
| level this rarely changes the label; at the level of narrower traditions it can.</p> | |
| <h2>Classifiers</h2> | |
| <p>For each head, a multinomial <strong>logistic regression</strong> (L-BFGS, balanced class weights, features standardised on the | |
| training split) is trained on the frozen embeddings. The regularisation strength <code>C</code> is selected on the validation split | |
| by macro-F1. Narrower-tradition heads are trained per broad region when at least two traditions have enough works; | |
| at inference a tradition is reported only when the broad-region head did not abstain.</p> | |
| <p><strong>Calibration.</strong> Raw classifier scores are not probabilities of historical correctness. A single temperature is fitted | |
| on validation logits by minimising negative log-likelihood (temperature scaling). Calibration quality is measured on the | |
| test split (expected calibration error, reliability diagrams).</p> | |
| <p><strong>Abstention.</strong> For each head, a threshold on the calibrated top score is chosen on the validation split as the lowest | |
| threshold whose selective accuracy reaches a target (80 %), subject to a minimum coverage. If the target cannot be met, | |
| the threshold enforces the minimum coverage and the interface says so. When a score is below the threshold the analysis | |
| reports <em>insufficient evidence</em>. Thresholds are never tuned on the test split.</p> | |
| <p><strong>Out-of-distribution flag.</strong> If an upload's nearest training image is less similar than the 1st percentile of | |
| validation images' nearest-training similarities, the result is marked as outside the model's well-evaluated scope. | |
| This is a heuristic based on distance in the embedding space, not a guarantee of detecting unfamiliar inputs.</p> | |
| <h2>Retrieval</h2> | |
| <p>The uploaded image's embedding is compared with every cached reference embedding by cosine similarity (an exact matrix | |
| product; no approximate index is needed for collections of this size). The top five results are returned, collapsing | |
| near-identical images (cosine ≥ 0.97 to an already listed result). Users can explicitly restrict results to the predicted | |
| region or period and switch back to the unfiltered ranking. The similarity-weighted labels of the ten nearest works are | |
| summarised next to the classifier output, and agreement or disagreement is reported.</p> | |
| <p>A similarity score measures resemblance under the DINOv2 representation. It is not evidence of a shared origin, | |
| workshop, or authenticity.</p> | |
| <h2>Evidence statements</h2> | |
| <p>The analysis page shows short statements such as "Retrieval disagrees: the 10 nearest reference works are mostly | |
| documented as …". They are produced by fixed rules in <code>src/artifex/predict.py</code> from the computed scores, thresholds, | |
| retrieval labels, training counts and the latest evaluation's per-class recall. No text is generated by a language model, | |
| and no statement refers to visual features (brushwork, pigments, motifs) that were not explicitly computed.</p> | |
| <h2>Evaluation</h2> | |
| <p>Summarised here; the full protocol is in <a href="#/docs/evaluation_protocol">evaluation_protocol.md</a>.</p> | |
| <ul> | |
| <li><strong>Grouping and splits.</strong> Works are grouped by artist (normalised name, across institutions), by accession lot for | |
| anonymous works and manuscript folios/album leaves, and by near-duplicate images (DINOv2 cosine ≥ 0.95). Whole groups | |
| are assigned to train/validation/test (70/15/15, stratified by region, fixed seed). At most 25 works per group enter | |
| training and evaluation so that a single manuscript cannot dominate a class.</li> | |
| <li><strong>Baselines.</strong> Majority class; logistic regression; linear SVM; similarity-weighted k-NN vote (retrieval only); | |
| the mean of logistic-regression and k-NN scores.</li> | |
| <li><strong>Metrics.</strong> Macro-F1, balanced accuracy, per-class precision/recall, confusion matrices; for period additionally | |
| within-one-bin accuracy, range-compatible accuracy and mean absolute error in years; calibration error; selective | |
| accuracy versus coverage; retrieval region/period precision@k and temporal closeness against a random-ranking baseline.</li> | |
| <li><strong>Generalisation.</strong> Artist-disjoint main split; collection-disjoint experiments (train on one museum, test on the | |
| other, removing test works whose artist or group appears in the training museum).</li> | |
| <li><strong>Ablations.</strong> Centre crop vs. whole image, greyscale vs. colour, DINOv2-S vs. DINOv2-B, colour/texture statistics.</li> | |
| </ul> | |
| <h2>Reproducing the experiments</h2> | |
| <pre><code class="language-bash">python scripts/ingest_data.py # fetch metadata (resumable; ~3 h for the Met at its rate limit) | |
| python scripts/build_dataset.py # normalise, download CC0 images, write the manifest | |
| python scripts/extract_embeddings.py --all-variants | |
| python scripts/train_models.py # splits + deployed heads | |
| python scripts/evaluate_models.py # metrics, report, artefacts for the website | |
| </code></pre> | |
| <p>Seeds, configuration snapshots, split assignments and model cards are saved with every evaluation run. Museum | |
| collections change over time, so re-ingesting later may produce a different dataset version; the manifest and split | |
| files of a run identify exactly which records were used.</p> | |
| <h2>Limitations</h2> | |
| <p>The main scientific, technical and historical limitations — collection bias, Eurocentric and court-centred holdings, | |
| cataloguing conventions, label noise, photographic confounds, unsupported traditions, and the meaning of the scores — | |
| are discussed in <a href="#/docs/limitations">limitations.md</a>. In short: Artifex estimates what is <em>typical of its reference | |
| collection</em>, not what is historically true of a given object.</p> |