Blame for data/docs/data_sources.html

<p>All reference data come from official museum open-access APIs. Nothing is scraped from web pages.</p>
<h2>The Metropolitan Museum of Art — Open Access API</h2>
<table>
<thead>
<tr>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>API</td>
<td><code>https://collectionapi.metmuseum.org/public/collection</code> — documentation <a href="https://metmuseum.github.io/">https://metmuseum.github.io/</a></td>
</tr>
<tr>
<td>Access</td>
<td>No key. The documentation states a limit of 80 requests/second, but during development (October 2026) the Met's firewall returned <strong>HTTP 403 after short bursts at 6 req/s and periodically at 1.5 req/s</strong> (~ every 2 minutes). Ingestion therefore runs at <strong>1.1 req/s</strong> with 60-second back-off on 403/429.</td>
</tr>
<tr>
<td>Endpoints</td>
<td><code>/v1.1/search</code> (paginated, max 500 per page — the legacy <code>/v1/search</code> was <strong>retired on 2026-10-01</strong>) to enumerate candidate object IDs per department; <code>/v1/objects/{id}</code> for full records.</td>
</tr>
<tr>
<td>Metadata licence</td>
<td>CC0 1.0 (Met Open Access policy).</td>
</tr>
<tr>
<td>Image licence</td>
<td>Images of objects flagged <code>isPublicDomain: true</code> are released under CC0 1.0. Artifex downloads <strong>only</strong> those (<code>primaryImageSmall</code>, the "web-large" derivative). Objects not in the public domain are excluded even when an image URL exists.</td>
</tr>
<tr>
<td>Attribution</td>
<td>Not legally required under CC0; Artifex records the credit line and links each work to its Met page.</td>
</tr>
</tbody>
</table>
<p>Candidate queries (<code>configs/dataset.yaml</code> → <code>sources.met.queries</code>): European Paintings (all), and <code>medium=Paintings</code>
in Asian Art, Islamic Art, Egyptian Art, Greek and Roman Art, Medieval Art, The Cloisters, Robert Lehman Collection,
Modern and Contemporary Art, Arts of Africa/Oceania/Americas, Ancient West Asian Art; plus Islamic Art <code>medium=Codices</code>,
filtered to records whose object name marks them as illustrated folios or paintings.</p>
<h2>The Cleveland Museum of Art — Open Access API</h2>
<table>
<thead>
<tr>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>API</td>
<td><code>https://openaccess-api.clevelandart.org/api/artworks/</code> — documentation <a href="https://openaccess-api.clevelandart.org/">https://openaccess-api.clevelandart.org/</a></td>
</tr>
<tr>
<td>Access</td>
<td>No key; paginated (<code>limit</code>/<code>skip</code>). Requests are rate-limited by Artifex to 6 req/s; images to 3 req/s per host.</td>
</tr>
<tr>
<td>Query</td>
<td><code>type=Painting&amp;has_image=1&amp;cc0=1</code>.</td>
</tr>
<tr>
<td>Metadata licence</td>
<td>CC0 1.0.</td>
</tr>
<tr>
<td>Image licence</td>
<td>CC0 1.0 for records with <code>share_license_status = "CC0"</code>; Artifex downloads the <code>images.web</code> derivative.</td>
</tr>
<tr>
<td>Attribution</td>
<td>Credit line recorded; each work links to its <code>clevelandart.org</code> page.</td>
</tr>
</tbody>
</table>
<p>Cleveland's <code>type=Manuscript</code> records (≈1,100) are <strong>not</strong> included in v0.1: many are text-only leaves and the
metadata do not reliably distinguish illustrated from text pages.</p>
<h2>Sources considered but not used</h2>
<ul>
<li><strong>Art Institute of Chicago API</strong> (<code>api.artic.edu</code>) — CC0 metadata and public-domain images, but during development
  both the API and the IIIF image server returned Cloudflare challenges (HTTP 503/403) to programmatic clients. Artifex
  does not attempt to bypass such protections. AIC's bulk data dump (S3) was reachable and is a candidate for a future
  version (images would still need the IIIF server).</li>
<li><strong>WikiArt and similar aggregations</strong> — image rights are unclear and labels are not institution-curated.</li>
</ul>
<h2>Exclusion rules (in order)</h2>
<ol>
<li>Not open access (Met: not public domain; Cleveland: not CC0).</li>
<li>No open-access image.</li>
<li>Met: classification not <em>Paintings</em> and object name not containing <em>illustrated</em> or <em>painting</em> and not exactly
   <em>Miniature</em>. (A substring match on "miniature" was used at first; it admitted 18 miniature stone vessels, a coffin
   model, a stela and jewellery from the Egyptian department, found while reviewing the collection and removed in the
   October 2026 build.)</li>
<li>Object name containing <em>photograph, print, reproduction, facsimile</em> (e.g. 20th-century facsimiles of Egyptian tomb
   paintings, which would carry ancient dates on modern images) or the word <em>palette</em> (painters' and scribes' palettes
   are tools, not painted imagery).</li>
<li>Title marking a text-only or non-pictorial page (<em>Text, folio …</em>, <em>Blank …</em>, <em>Calligraphy …</em>).</li>
<li>Image download or validation failure (e.g. web derivatives of handscrolls narrower than 64 px).</li>
<li>Exact duplicates (same image URL or identical bytes) are kept in the manifest but marked <code>duplicate_of</code> and excluded
   from the index and modelling.</li>
</ol>
<p>Every exclusion is recorded with its reason in <code>data/manifests/exclusions.csv</code>.</p>
<h2>Provenance kept per record</h2>
<p><code>artwork_id</code> (<code>met:&lt;objectID&gt;</code>, <code>cma:&lt;id&gt;</code>), <code>source_object_id</code>, <code>source_url</code>, <code>accession_number</code>, <code>image_url</code>,
<code>image_license</code>, <code>metadata_license</code>, <code>license_attribution</code> (credit line), <code>ingestion_timestamp</code> (time of the API
response), <code>dataset_version</code>. Raw API responses are kept verbatim under <code>data/raw/</code> (not committed).</p>
<h2>Reproducing</h2>
<pre><code class="language-bash">python scripts/ingest_data.py          # resumable; re-run to fill gaps after failures
python scripts/build_dataset.py
</code></pre>
<p>Failed object requests are logged to <code>data/raw/met/fetch_failures.jsonl</code> and retried on the next run; image results to
<code>data/processed/image_log.jsonl</code>. In the October 2026 build, 355 of 13,120 Met search hits (2.7 %) returned
<code>404 Not Found</code> from the object endpoint on every attempt — the search index lists some objects the object endpoint
does not serve. They are simply absent from the dataset.</p>