Exploring the visual signatures of art across time and place.

Artifex is a locally runnable computer-vision research application that estimates the likely cultural-geographic region and historical period of a painting from its image alone, retrieves the most similar works from a documented reference collection of open-access museum records, and shows the evidence, uncertainty and measured limits behind every estimate.

It is a personal research prototype in computational art history. It does not authenticate or attribute artworks.

No large language models, no paid APIs, no API keys. Uploaded images are analysed in memory on your machine.
Image representation: frozen DINOv2 (ViT-B/14, Apache-2.0). Classifiers: logistic regression with temperature scaling and abstention. Retrieval: exact cosine similarity.
Reference data: CC0 records and images from The Metropolitan Museum of Art and The Cleveland Museum of Art.
Every metric shown in the app is read from evaluation artefacts produced by the code in this repository.

Try it: https://sucksome-artifex.static.hf.space — the in-browser edition. The same model runs in your browser; uploaded images never leave your device. See docs/deployment.md.

Measured results (latest run)

Generated from artifacts/evaluation/runs/20261022T054811Z_e94d93bf34/metrics.json — model artifex-0.1-e94d93bf34, dataset 0.1+5cc3c8d20c19. Test split: 1,352 works whose artists, manuscripts/albums and near-duplicates do not occur in training.

Task	Classes	Logistic regression macro-F1 [95% CI]	Balanced acc.	k-NN retrieval macro-F1	Majority macro-F1
region	7	0.705 [0.64–0.76]	0.716	0.658	0.077
period	9	0.420 [0.36–0.48]	0.444	0.397	0.050

Period predictions fall within one bin of the labelled bin for 78.2% of test works (mean distance to the documented date range 139 years).

Collection-disjoint (train on one museum, test on the other): met → cleveland: region macro-F1 0.816 over 4 shared classes; cleveland → met: region macro-F1 0.662 over 5 shared classes.

Full report: artifacts/evaluation/runs/20261022T054811Z_e94d93bf34/report.md and the Experiments page. These numbers describe these two collections only; see docs/limitations.md.

Path	What it is
`src/artifex/`	the pipeline: ingestion, normalisation, labels, images, features, splits, training, retrieval, inference, evaluation
`scripts/`	command-line entry points (below)
`app/`	the Streamlit website (`app/main.py`, pages in `app/views/`)
`web/`	the in-browser edition (static HTML/JavaScript; the same model runs in the visitor's browser via ONNX Runtime Web)
`configs/`	every tunable choice: sources and filters, period bins, region taxonomy, preprocessing, classifiers, evaluation
`data/manifests/`	the versioned manifest, exclusions, region-mapping report, dataset statistics
`data/splits/`	split assignments
`artifacts/`	embeddings, trained heads, evaluation runs (generated)
`docs/`	research question, methodology, data sources, dataset card, taxonomy, evaluation protocol, limitations, deployment, roadmap
`tests/`	unit, integration and headless UI tests

Installation

Requires Python 3.10–3.12. A CUDA GPU is optional.

python -m venv .venv
# Windows: .venv\Scripts\activate    macOS/Linux: source .venv/bin/activate
pip install -e ".[dev]"

For GPU acceleration install a CUDA build of PyTorch first, e.g. pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124.

Building everything (in order)

Step	Command	Produces	Time on the development machine
1. Ingest metadata	`python scripts/ingest_data.py`	raw API responses in `data/raw/`	Cleveland ≈ 2 min; the Met ≈ 3.5 h (its firewall limits clients to ≈ 1 request/s)
2. Build dataset	`python scripts/build_dataset.py`	images in `data/processed/`, `data/manifests/*`	≈ 1 h of image downloads
3. Embeddings	`python scripts/extract_embeddings.py` (add `--all-variants` for ablations)	`artifacts/embeddings/<variant>/`	≈ 2 min per DINOv2 variant for ~10,000 images on an RTX 3050 Laptop GPU (≈ 11 ms per image)
4. Train	`python scripts/train_models.py`	`data/splits/`, `artifacts/classifiers/deployed/`	≈ 1 min
5. Evaluate	`python scripts/evaluate_models.py`	`artifacts/evaluation/runs/<run>/`, `latest.json`	≈ 3.5 min with ablations (measured 200 s on a Ryzen 7 5800HS); see the run's `runtime_seconds`
6. Launch	`streamlit run app/main.py`	http://localhost:8501

In-browser edition (static website, no server; see docs/deployment.md): python scripts/export_browser_model.py, then python scripts/build_static_site.py, python scripts/validate_browser.py (compares the browser pipeline with the Python application in headless Chromium) and python scripts/build_static_site.py --no-thumbnails to include that comparison; preview with python scripts/serve_static.py.

All steps are resumable or idempotent: ingestion skips cached objects, image downloads skip existing files, embedding extraction reuses cached rows. Analyse one image from the command line with python scripts/run_demo.py path/to/image.jpg.

What is downloaded, generated, or not redistributed

Resource	How you get it
DINOv2 weights (`facebook/dinov2-base`, `-small`)	downloaded automatically from the Hugging Face Hub on first use, pinned to a commit (`configs/preprocessing.yaml`) into `models/hf/`
Museum metadata and images	fetched by steps 1–2 from the museums' official open-access APIs (CC0); not committed to Git
Manifest, splits, statistics	generated (small; can be committed for reproducibility)
Embeddings, classifier heads, evaluation runs	generated by steps 3–5

The website shows explicit "dataset/model unavailable" and "no evaluation yet" states until the corresponding steps have run; it never shows placeholder predictions.

Using the website

Home — what Artifex is, a sample of the collection, and the headline evaluation numbers of the latest run.
Analyse — upload a JPEG/PNG/WebP (≤ 20 MB). You get ranked region and period estimates with calibrated scores and abstention ("insufficient evidence"), a narrower tradition where supported, the five most similar reference works (with an explicit, reversible filter by predicted region or period), what the ten nearest works are documented as, and rule-based evidence statements (agreement/disagreement, training coverage, held-out performance for the predicted class, near-duplicate and out-of-distribution warnings). A JSON copy of the result can be downloaded.
Collection — search by title/artist/institution, filter by region, period, collection and medium, sort, page, and open a full record with provenance, licence, metadata caveats and visually closest works.
Methodology — the method (docs/methodology.md) and the live configuration: model version, encoder commit, classes, thresholds, bins, taxonomy and the full region-mapping report.
Experiments — every metric of the latest evaluation run: model comparisons with baselines, per-class tables, confusion matrices, calibration and risk–coverage curves, retrieval vs. random, collection-disjoint results, ablations, a gallery of confident mistakes and dataset composition; plus downloads of metrics.json, predictions.csv, report.md.
About — motivation, data and licence policy, privacy, stack, citation.

Screenshots of the deployed in-browser edition (model artifex-0.1-765e3b9a93, headless Chromium at 1440 px and 390 px; the local Streamlit app has the same pages and content) are in docs/screenshots/: the analysis of a held-out test image (desktop, mobile), the experiments, collection, record, home and methodology pages. The analysed image is itself in the reference collection, so it retrieves itself at similarity 1.00 and Artifex warns that the scores are not independent evidence. Artwork images in the screenshots come from the two museums' CC0 open-access releases.

Tests

pytest                  # unit + integration + headless UI tests (offline, ~1–2 min)
pytest -m slow          # real DINOv2 encoder tests (download the checkpoint on first use)

The integration and UI tests run the real pipeline end to end on a small synthetic fixture (procedurally generated images with artificial labels) using the non-learned colour/texture encoder, so they need neither network access nor model downloads. The fixture exists only to test software; it is never presented as evidence.

Troubleshooting

Symptom	Cause / fix
Met ingestion logs `throttled (403)`	The Met's firewall rate-limits bursts. The client backs off automatically; lower `sources.met.http.requests_per_second` if it persists. Re-running resumes.
"The dataset changed after the model was trained"	The manifest was rebuilt. Run steps 3–5 again.
"Cached embeddings do not match the trained model"	Preprocessing or encoder config changed. Re-run steps 3–5.
Analysis page says model or dataset unavailable	One of steps 1–4 has not been run; the page lists which.
Slow first analysis	The encoder loads once per app process (≈ 10 s).
No GPU used	Install a CUDA build of PyTorch; `--device cpu` forces CPU.
Art Institute of Chicago	Not used: its API/IIIF server blocked programmatic clients during development (see `docs/data_sources.md`).

Documentation

Research question · Methodology · Data sources · Dataset card · Label taxonomy · Annotation protocol · Evaluation protocol · Limitations · Deployment · Roadmap

Licence and acknowledgements

Code: MIT (LICENSE). Reference data and images: The Metropolitan Museum of Art Open Access and The Cleveland Museum of Art Open Access, CC0 1.0. DINOv2: Meta AI, Apache-2.0. See CITATION.cff for citation.