All reference data come from official museum open-access APIs. Nothing is scraped from web pages.

The Metropolitan Museum of Art — Open Access API


API	`https://collectionapi.metmuseum.org/public/collection` — documentation https://metmuseum.github.io/
Access	No key. The documentation states a limit of 80 requests/second, but during development (October 2026) the Met's firewall returned HTTP 403 after short bursts at 6 req/s and periodically at 1.5 req/s (~ every 2 minutes). Ingestion therefore runs at 1.1 req/s with 60-second back-off on 403/429.
Endpoints	`/v1.1/search` (paginated, max 500 per page — the legacy `/v1/search` was retired on 2026-10-01) to enumerate candidate object IDs per department; `/v1/objects/{id}` for full records.
Metadata licence	CC0 1.0 (Met Open Access policy).
Image licence	Images of objects flagged `isPublicDomain: true` are released under CC0 1.0. Artifex downloads only those (`primaryImageSmall`, the "web-large" derivative). Objects not in the public domain are excluded even when an image URL exists.
Attribution	Not legally required under CC0; Artifex records the credit line and links each work to its Met page.

Candidate queries (configs/dataset.yaml → sources.met.queries): European Paintings (all), and medium=Paintings in Asian Art, Islamic Art, Egyptian Art, Greek and Roman Art, Medieval Art, The Cloisters, Robert Lehman Collection, Modern and Contemporary Art, Arts of Africa/Oceania/Americas, Ancient West Asian Art; plus Islamic Art medium=Codices, filtered to records whose object name marks them as illustrated folios or paintings.

The Cleveland Museum of Art — Open Access API


API	`https://openaccess-api.clevelandart.org/api/artworks/` — documentation https://openaccess-api.clevelandart.org/
Access	No key; paginated (`limit`/`skip`). Requests are rate-limited by Artifex to 6 req/s; images to 3 req/s per host.
Query	`type=Painting&has_image=1&cc0=1`.
Metadata licence	CC0 1.0.
Image licence	CC0 1.0 for records with `share_license_status = "CC0"`; Artifex downloads the `images.web` derivative.
Attribution	Credit line recorded; each work links to its `clevelandart.org` page.

Cleveland's type=Manuscript records (≈1,100) are not included in v0.1: many are text-only leaves and the metadata do not reliably distinguish illustrated from text pages.

Sources considered but not used

Art Institute of Chicago API (api.artic.edu) — CC0 metadata and public-domain images, but during development both the API and the IIIF image server returned Cloudflare challenges (HTTP 503/403) to programmatic clients. Artifex does not attempt to bypass such protections. AIC's bulk data dump (S3) was reachable and is a candidate for a future version (images would still need the IIIF server).
WikiArt and similar aggregations — image rights are unclear and labels are not institution-curated.

Exclusion rules (in order)

Not open access (Met: not public domain; Cleveland: not CC0).
No open-access image.
Met: classification not Paintings and object name not containing illustrated or painting and not exactly Miniature. (A substring match on "miniature" was used at first; it admitted 18 miniature stone vessels, a coffin model, a stela and jewellery from the Egyptian department, found while reviewing the collection and removed in the October 2026 build.)
Object name containing photograph, print, reproduction, facsimile (e.g. 20th-century facsimiles of Egyptian tomb paintings, which would carry ancient dates on modern images) or the word palette (painters' and scribes' palettes are tools, not painted imagery).
Title marking a text-only or non-pictorial page (Text, folio …, Blank …, Calligraphy …).
Image download or validation failure (e.g. web derivatives of handscrolls narrower than 64 px).
Exact duplicates (same image URL or identical bytes) are kept in the manifest but marked duplicate_of and excluded from the index and modelling.

Every exclusion is recorded with its reason in data/manifests/exclusions.csv.

Provenance kept per record

artwork_id (met:<objectID>, cma:<id>), source_object_id, source_url, accession_number, image_url, image_license, metadata_license, license_attribution (credit line), ingestion_timestamp (time of the API response), dataset_version. Raw API responses are kept verbatim under data/raw/ (not committed).

Reproducing

python scripts/ingest_data.py          # resumable; re-run to fill gaps after failures
python scripts/build_dataset.py

Failed object requests are logged to data/raw/met/fetch_failures.jsonl and retried on the next run; image results to data/processed/image_log.jsonl. In the October 2026 build, 355 of 13,120 Met search hits (2.7 %) returned 404 Not Found from the object endpoint on every attempt — the search index lists some objects the object endpoint does not serve. They are simply absent from the dataset.