Blame for data/docs/readme.html

<p><em>Exploring the visual signatures of art across time and place.</em></p>
<p>Artifex is a locally runnable computer-vision research application that estimates the likely <strong>cultural-geographic
region</strong> and <strong>historical period</strong> of a painting from its image alone, retrieves the most similar works from a documented
reference collection of open-access museum records, and shows the evidence, uncertainty and measured limits behind every
estimate.</p>
<p>It is a personal research prototype in computational art history. It does <strong>not</strong> authenticate or attribute artworks.</p>
<ul>
<li>No large language models, no paid APIs, no API keys. Uploaded images are analysed in memory on your machine.</li>
<li>Image representation: frozen <strong>DINOv2</strong> (ViT-B/14, Apache-2.0). Classifiers: logistic regression with temperature
  scaling and abstention. Retrieval: exact cosine similarity.</li>
<li>Reference data: CC0 records and images from <strong>The Metropolitan Museum of Art</strong> and <strong>The Cleveland Museum of Art</strong>.</li>
<li>Every metric shown in the app is read from evaluation artefacts produced by the code in this repository.</li>
</ul>
<p><strong>Try it:</strong> <a href="https://sucksome-artifex.static.hf.space">https://sucksome-artifex.static.hf.space</a> — the in-browser edition. The same model runs in your browser;
uploaded images never leave your device. See <a href="#/docs/deployment">docs/deployment.md</a>.</p>
<!-- RESULTS:START -->
<h2>Measured results (latest run)</h2>
<p><em>Generated from <code>artifacts/evaluation/runs/20261022T054811Z_e94d93bf34/metrics.json</code> — model <code>artifex-0.1-e94d93bf34</code>, dataset <code>0.1+5cc3c8d20c19</code>. Test split: 1,352 works whose artists, manuscripts/albums and near-duplicates do not occur in training.</em></p>
<table>
<thead>
<tr>
<th>Task</th>
<th>Classes</th>
<th>Logistic regression macro-F1 [95% CI]</th>
<th>Balanced acc.</th>
<th>k-NN retrieval macro-F1</th>
<th>Majority macro-F1</th>
</tr>
</thead>
<tbody>
<tr>
<td>region</td>
<td>7</td>
<td>0.705 [0.64–0.76]</td>
<td>0.716</td>
<td>0.658</td>
<td>0.077</td>
</tr>
<tr>
<td>period</td>
<td>9</td>
<td>0.420 [0.36–0.48]</td>
<td>0.444</td>
<td>0.397</td>
<td>0.050</td>
</tr>
</tbody>
</table>
<p>Period predictions fall within one bin of the labelled bin for 78.2% of test works (mean distance to the documented date range 139 years).</p>
<p>Collection-disjoint (train on one museum, test on the other): met → cleveland: region macro-F1 0.816 over 4 shared classes; cleveland → met: region macro-F1 0.662 over 5 shared classes.</p>
<p>Full report: <code>artifacts/evaluation/runs/20261022T054811Z_e94d93bf34/report.md</code> and the Experiments page. These numbers describe these two collections only; see <code>docs/limitations.md</code>.</p>
<!-- RESULTS:END -->

<h2>Contents</h2>
<table>
<thead>
<tr>
<th>Path</th>
<th>What it is</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>src/artifex/</code></td>
<td>the pipeline: ingestion, normalisation, labels, images, features, splits, training, retrieval, inference, evaluation</td>
</tr>
<tr>
<td><code>scripts/</code></td>
<td>command-line entry points (below)</td>
</tr>
<tr>
<td><code>app/</code></td>
<td>the Streamlit website (<code>app/main.py</code>, pages in <code>app/views/</code>)</td>
</tr>
<tr>
<td><code>web/</code></td>
<td>the in-browser edition (static HTML/JavaScript; the same model runs in the visitor's browser via ONNX Runtime Web)</td>
</tr>
<tr>
<td><code>configs/</code></td>
<td>every tunable choice: sources and filters, period bins, region taxonomy, preprocessing, classifiers, evaluation</td>
</tr>
<tr>
<td><code>data/manifests/</code></td>
<td>the versioned manifest, exclusions, region-mapping report, dataset statistics</td>
</tr>
<tr>
<td><code>data/splits/</code></td>
<td>split assignments</td>
</tr>
<tr>
<td><code>artifacts/</code></td>
<td>embeddings, trained heads, evaluation runs (generated)</td>
</tr>
<tr>
<td><code>docs/</code></td>
<td>research question, methodology, data sources, dataset card, taxonomy, evaluation protocol, limitations, deployment, roadmap</td>
</tr>
<tr>
<td><code>tests/</code></td>
<td>unit, integration and headless UI tests</td>
</tr>
</tbody>
</table>
<h2>Installation</h2>
<p>Requires Python 3.10–3.12. A CUDA GPU is optional.</p>
<pre><code class="language-bash">python -m venv .venv
# Windows: .venv\Scripts\activate    macOS/Linux: source .venv/bin/activate
pip install -e &quot;.[dev]&quot;
</code></pre>
<p>For GPU acceleration install a CUDA build of PyTorch first, e.g.
<code>pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124</code>.</p>
<h2>Building everything (in order)</h2>
<table>
<thead>
<tr>
<th>Step</th>
<th>Command</th>
<th>Produces</th>
<th>Time on the development machine</th>
</tr>
</thead>
<tbody>
<tr>
<td>1. Ingest metadata</td>
<td><code>python scripts/ingest_data.py</code></td>
<td>raw API responses in <code>data/raw/</code></td>
<td>Cleveland ≈ 2 min; the Met ≈ 3.5 h (its firewall limits clients to ≈ 1 request/s)</td>
</tr>
<tr>
<td>2. Build dataset</td>
<td><code>python scripts/build_dataset.py</code></td>
<td>images in <code>data/processed/</code>, <code>data/manifests/*</code></td>
<td>≈ 1 h of image downloads</td>
</tr>
<tr>
<td>3. Embeddings</td>
<td><code>python scripts/extract_embeddings.py</code> (add <code>--all-variants</code> for ablations)</td>
<td><code>artifacts/embeddings/&lt;variant&gt;/</code></td>
<td>≈ 2 min per DINOv2 variant for ~10,000 images on an RTX 3050 Laptop GPU (≈ 11 ms per image)</td>
</tr>
<tr>
<td>4. Train</td>
<td><code>python scripts/train_models.py</code></td>
<td><code>data/splits/</code>, <code>artifacts/classifiers/deployed/</code></td>
<td>≈ 1 min</td>
</tr>
<tr>
<td>5. Evaluate</td>
<td><code>python scripts/evaluate_models.py</code></td>
<td><code>artifacts/evaluation/runs/&lt;run&gt;/</code>, <code>latest.json</code></td>
<td>≈ 3.5 min with ablations (measured 200 s on a Ryzen 7 5800HS); see the run's <code>runtime_seconds</code></td>
</tr>
<tr>
<td>6. Launch</td>
<td><code>streamlit run app/main.py</code></td>
<td><a href="http://localhost:8501">http://localhost:8501</a></td>
<td></td>
</tr>
</tbody>
</table>
<p><strong>In-browser edition</strong> (static website, no server; see <a href="#/docs/deployment">docs/deployment.md</a>):
<code>python scripts/export_browser_model.py</code>, then <code>python scripts/build_static_site.py</code>, <code>python scripts/validate_browser.py</code>
(compares the browser pipeline with the Python application in headless Chromium) and <code>python scripts/build_static_site.py
--no-thumbnails</code> to include that comparison; preview with <code>python scripts/serve_static.py</code>.</p>
<p>All steps are resumable or idempotent: ingestion skips cached objects, image downloads skip existing files, embedding
extraction reuses cached rows. Analyse one image from the command line with <code>python scripts/run_demo.py path/to/image.jpg</code>.</p>
<h3>What is downloaded, generated, or not redistributed</h3>
<table>
<thead>
<tr>
<th>Resource</th>
<th>How you get it</th>
</tr>
</thead>
<tbody>
<tr>
<td>DINOv2 weights (<code>facebook/dinov2-base</code>, <code>-small</code>)</td>
<td>downloaded automatically from the Hugging Face Hub on first use, pinned to a commit (<code>configs/preprocessing.yaml</code>) into <code>models/hf/</code></td>
</tr>
<tr>
<td>Museum metadata and images</td>
<td>fetched by steps 1–2 from the museums' official open-access APIs (CC0); not committed to Git</td>
</tr>
<tr>
<td>Manifest, splits, statistics</td>
<td>generated (small; can be committed for reproducibility)</td>
</tr>
<tr>
<td>Embeddings, classifier heads, evaluation runs</td>
<td>generated by steps 3–5</td>
</tr>
</tbody>
</table>
<p>The website shows explicit "dataset/model unavailable" and "no evaluation yet" states until the corresponding steps have
run; it never shows placeholder predictions.</p>
<h2>Using the website</h2>
<ul>
<li><strong>Home</strong> — what Artifex is, a sample of the collection, and the headline evaluation numbers of the latest run.</li>
<li><strong>Analyse</strong> — upload a JPEG/PNG/WebP (≤ 20 MB). You get ranked region and period estimates with calibrated scores and
  abstention ("insufficient evidence"), a narrower tradition where supported, the five most similar reference works (with
  an explicit, reversible filter by predicted region or period), what the ten nearest works are documented as, and
  rule-based evidence statements (agreement/disagreement, training coverage, held-out performance for the predicted
  class, near-duplicate and out-of-distribution warnings). A JSON copy of the result can be downloaded.</li>
<li><strong>Collection</strong> — search by title/artist/institution, filter by region, period, collection and medium, sort, page, and
  open a full record with provenance, licence, metadata caveats and visually closest works.</li>
<li><strong>Methodology</strong> — the method (<code>docs/methodology.md</code>) and the live configuration: model version, encoder commit,
  classes, thresholds, bins, taxonomy and the full region-mapping report.</li>
<li><strong>Experiments</strong> — every metric of the latest evaluation run: model comparisons with baselines, per-class tables,
  confusion matrices, calibration and risk–coverage curves, retrieval vs. random, collection-disjoint results, ablations,
  a gallery of confident mistakes and dataset composition; plus downloads of <code>metrics.json</code>, <code>predictions.csv</code>, <code>report.md</code>.</li>
<li><strong>About</strong> — motivation, data and licence policy, privacy, stack, citation.</li>
</ul>
<p>Screenshots of the deployed in-browser edition (model <code>artifex-0.1-765e3b9a93</code>, headless Chromium at 1440 px and
390 px; the local Streamlit app has the same pages and content) are in <a href="docs/screenshots/desktop_analyse.jpg"><code>docs/screenshots/</code></a>: the
analysis of a held-out test image (<a href="docs/screenshots/desktop_analyse.jpg">desktop</a>,
<a href="docs/screenshots/mobile_analyse.jpg">mobile</a>), the <a href="docs/screenshots/desktop_experiments.jpg">experiments</a>,
<a href="docs/screenshots/desktop_collection.jpg">collection</a>, <a href="docs/screenshots/desktop_collection_dialog.jpg">record</a>,
<a href="docs/screenshots/desktop_home.jpg">home</a> and <a href="docs/screenshots/desktop_methodology.jpg">methodology</a> pages. The analysed
image is itself in the reference collection, so it retrieves itself at similarity 1.00 and Artifex warns that the scores
are not independent evidence. Artwork images in the screenshots come from the two museums' CC0 open-access releases.</p>
<h2>Tests</h2>
<pre><code class="language-bash">pytest                  # unit + integration + headless UI tests (offline, ~1–2 min)
pytest -m slow          # real DINOv2 encoder tests (download the checkpoint on first use)
</code></pre>
<p>The integration and UI tests run the real pipeline end to end on a small <strong>synthetic</strong> fixture (procedurally generated
images with artificial labels) using the non-learned colour/texture encoder, so they need neither network access nor
model downloads. The fixture exists only to test software; it is never presented as evidence.</p>
<h2>Troubleshooting</h2>
<table>
<thead>
<tr>
<th>Symptom</th>
<th>Cause / fix</th>
</tr>
</thead>
<tbody>
<tr>
<td>Met ingestion logs <code>throttled (403)</code></td>
<td>The Met's firewall rate-limits bursts. The client backs off automatically; lower <code>sources.met.http.requests_per_second</code> if it persists. Re-running resumes.</td>
</tr>
<tr>
<td>"The dataset changed after the model was trained"</td>
<td>The manifest was rebuilt. Run steps 3–5 again.</td>
</tr>
<tr>
<td>"Cached embeddings do not match the trained model"</td>
<td>Preprocessing or encoder config changed. Re-run steps 3–5.</td>
</tr>
<tr>
<td>Analysis page says model or dataset unavailable</td>
<td>One of steps 1–4 has not been run; the page lists which.</td>
</tr>
<tr>
<td>Slow first analysis</td>
<td>The encoder loads once per app process (≈ 10 s).</td>
</tr>
<tr>
<td>No GPU used</td>
<td>Install a CUDA build of PyTorch; <code>--device cpu</code> forces CPU.</td>
</tr>
<tr>
<td>Art Institute of Chicago</td>
<td>Not used: its API/IIIF server blocked programmatic clients during development (see <code>docs/data_sources.md</code>).</td>
</tr>
</tbody>
</table>
<h2>Documentation</h2>
<p><a href="#/docs/research_question">Research question</a> · <a href="#/docs/methodology">Methodology</a> ·
<a href="#/docs/data_sources">Data sources</a> · <a href="#/docs/dataset_card">Dataset card</a> · <a href="#/docs/label_taxonomy">Label taxonomy</a> ·
<a href="#/docs/annotation_protocol">Annotation protocol</a> · <a href="#/docs/evaluation_protocol">Evaluation protocol</a> ·
<a href="#/docs/limitations">Limitations</a> · <a href="#/docs/deployment">Deployment</a> · <a href="#/docs/roadmap">Roadmap</a></p>
<h2>Licence and acknowledgements</h2>
<p>Code: MIT (<code>LICENSE</code>). Reference data and images: The Metropolitan Museum of Art Open Access and The Cleveland Museum of
Art Open Access, CC0 1.0. DINOv2: Meta AI, Apache-2.0. See <code>CITATION.cff</code> for citation.</p>