| <p>All reference data come from official museum open-access APIs. Nothing is scraped from web pages.</p> | |
| <h2>The Metropolitan Museum of Art — Open Access API</h2> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th></th> | |
| <th></th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>API</td> | |
| <td><code>https://collectionapi.metmuseum.org/public/collection</code> — documentation <a href="https://metmuseum.github.io/">https://metmuseum.github.io/</a></td> | |
| </tr> | |
| <tr> | |
| <td>Access</td> | |
| <td>No key. The documentation states a limit of 80 requests/second, but during development (October 2026) the Met's firewall returned <strong>HTTP 403 after short bursts at 6 req/s and periodically at 1.5 req/s</strong> (~ every 2 minutes). Ingestion therefore runs at <strong>1.1 req/s</strong> with 60-second back-off on 403/429.</td> | |
| </tr> | |
| <tr> | |
| <td>Endpoints</td> | |
| <td><code>/v1.1/search</code> (paginated, max 500 per page — the legacy <code>/v1/search</code> was <strong>retired on 2026-10-01</strong>) to enumerate candidate object IDs per department; <code>/v1/objects/{id}</code> for full records.</td> | |
| </tr> | |
| <tr> | |
| <td>Metadata licence</td> | |
| <td>CC0 1.0 (Met Open Access policy).</td> | |
| </tr> | |
| <tr> | |
| <td>Image licence</td> | |
| <td>Images of objects flagged <code>isPublicDomain: true</code> are released under CC0 1.0. Artifex downloads <strong>only</strong> those (<code>primaryImageSmall</code>, the "web-large" derivative). Objects not in the public domain are excluded even when an image URL exists.</td> | |
| </tr> | |
| <tr> | |
| <td>Attribution</td> | |
| <td>Not legally required under CC0; Artifex records the credit line and links each work to its Met page.</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p>Candidate queries (<code>configs/dataset.yaml</code> → <code>sources.met.queries</code>): European Paintings (all), and <code>medium=Paintings</code> | |
| in Asian Art, Islamic Art, Egyptian Art, Greek and Roman Art, Medieval Art, The Cloisters, Robert Lehman Collection, | |
| Modern and Contemporary Art, Arts of Africa/Oceania/Americas, Ancient West Asian Art; plus Islamic Art <code>medium=Codices</code>, | |
| filtered to records whose object name marks them as illustrated folios or paintings.</p> | |
| <h2>The Cleveland Museum of Art — Open Access API</h2> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th></th> | |
| <th></th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>API</td> | |
| <td><code>https://openaccess-api.clevelandart.org/api/artworks/</code> — documentation <a href="https://openaccess-api.clevelandart.org/">https://openaccess-api.clevelandart.org/</a></td> | |
| </tr> | |
| <tr> | |
| <td>Access</td> | |
| <td>No key; paginated (<code>limit</code>/<code>skip</code>). Requests are rate-limited by Artifex to 6 req/s; images to 3 req/s per host.</td> | |
| </tr> | |
| <tr> | |
| <td>Query</td> | |
| <td><code>type=Painting&has_image=1&cc0=1</code>.</td> | |
| </tr> | |
| <tr> | |
| <td>Metadata licence</td> | |
| <td>CC0 1.0.</td> | |
| </tr> | |
| <tr> | |
| <td>Image licence</td> | |
| <td>CC0 1.0 for records with <code>share_license_status = "CC0"</code>; Artifex downloads the <code>images.web</code> derivative.</td> | |
| </tr> | |
| <tr> | |
| <td>Attribution</td> | |
| <td>Credit line recorded; each work links to its <code>clevelandart.org</code> page.</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p>Cleveland's <code>type=Manuscript</code> records (≈1,100) are <strong>not</strong> included in v0.1: many are text-only leaves and the | |
| metadata do not reliably distinguish illustrated from text pages.</p> | |
| <h2>Sources considered but not used</h2> | |
| <ul> | |
| <li><strong>Art Institute of Chicago API</strong> (<code>api.artic.edu</code>) — CC0 metadata and public-domain images, but during development | |
| both the API and the IIIF image server returned Cloudflare challenges (HTTP 503/403) to programmatic clients. Artifex | |
| does not attempt to bypass such protections. AIC's bulk data dump (S3) was reachable and is a candidate for a future | |
| version (images would still need the IIIF server).</li> | |
| <li><strong>WikiArt and similar aggregations</strong> — image rights are unclear and labels are not institution-curated.</li> | |
| </ul> | |
| <h2>Exclusion rules (in order)</h2> | |
| <ol> | |
| <li>Not open access (Met: not public domain; Cleveland: not CC0).</li> | |
| <li>No open-access image.</li> | |
| <li>Met: classification not <em>Paintings</em> and object name not containing <em>illustrated</em> or <em>painting</em> and not exactly | |
| <em>Miniature</em>. (A substring match on "miniature" was used at first; it admitted 18 miniature stone vessels, a coffin | |
| model, a stela and jewellery from the Egyptian department, found while reviewing the collection and removed in the | |
| October 2026 build.)</li> | |
| <li>Object name containing <em>photograph, print, reproduction, facsimile</em> (e.g. 20th-century facsimiles of Egyptian tomb | |
| paintings, which would carry ancient dates on modern images) or the word <em>palette</em> (painters' and scribes' palettes | |
| are tools, not painted imagery).</li> | |
| <li>Title marking a text-only or non-pictorial page (<em>Text, folio …</em>, <em>Blank …</em>, <em>Calligraphy …</em>).</li> | |
| <li>Image download or validation failure (e.g. web derivatives of handscrolls narrower than 64 px).</li> | |
| <li>Exact duplicates (same image URL or identical bytes) are kept in the manifest but marked <code>duplicate_of</code> and excluded | |
| from the index and modelling.</li> | |
| </ol> | |
| <p>Every exclusion is recorded with its reason in <code>data/manifests/exclusions.csv</code>.</p> | |
| <h2>Provenance kept per record</h2> | |
| <p><code>artwork_id</code> (<code>met:<objectID></code>, <code>cma:<id></code>), <code>source_object_id</code>, <code>source_url</code>, <code>accession_number</code>, <code>image_url</code>, | |
| <code>image_license</code>, <code>metadata_license</code>, <code>license_attribution</code> (credit line), <code>ingestion_timestamp</code> (time of the API | |
| response), <code>dataset_version</code>. Raw API responses are kept verbatim under <code>data/raw/</code> (not committed).</p> | |
| <h2>Reproducing</h2> | |
| <pre><code class="language-bash">python scripts/ingest_data.py # resumable; re-run to fill gaps after failures | |
| python scripts/build_dataset.py | |
| </code></pre> | |
| <p>Failed object requests are logged to <code>data/raw/met/fetch_failures.jsonl</code> and retried on the next run; image results to | |
| <code>data/processed/image_log.jsonl</code>. In the October 2026 build, 355 of 13,120 Met search hits (2.7 %) returned | |
| <code>404 Not Found</code> from the object endpoint on every attempt — the search index lists some objects the object endpoint | |
| does not serve. They are simply absent from the dataset.</p> |