| <p><em>Exploring the visual signatures of art across time and place.</em></p>
|
| <p>Artifex is a locally runnable computer-vision research application that estimates the likely <strong>cultural-geographic
|
| region</strong> and <strong>historical period</strong> of a painting from its image alone, retrieves the most similar works from a documented
|
| reference collection of open-access museum records, and shows the evidence, uncertainty and measured limits behind every
|
| estimate.</p>
|
| <p>It is a personal research prototype in computational art history. It does <strong>not</strong> authenticate or attribute artworks.</p>
|
| <ul>
|
| <li>No large language models, no paid APIs, no API keys. Uploaded images are analysed in memory on your machine.</li>
|
| <li>Image representation: frozen <strong>DINOv2</strong> (ViT-B/14, Apache-2.0). Classifiers: logistic regression with temperature
|
| scaling and abstention. Retrieval: exact cosine similarity.</li>
|
| <li>Reference data: CC0 records and images from <strong>The Metropolitan Museum of Art</strong> and <strong>The Cleveland Museum of Art</strong>.</li>
|
| <li>Every metric shown in the app is read from evaluation artefacts produced by the code in this repository.</li>
|
| </ul>
|
| <p><strong>Try it:</strong> <a href="https://sucksome-artifex.static.hf.space">https://sucksome-artifex.static.hf.space</a> — the in-browser edition. The same model runs in your browser;
|
| uploaded images never leave your device. See <a href="#/docs/deployment">docs/deployment.md</a>.</p>
|
|
|
| <h2>Measured results (latest run)</h2>
|
| <p><em>Generated from <code>artifacts/evaluation/runs/20261022T054811Z_e94d93bf34/metrics.json</code> — model <code>artifex-0.1-e94d93bf34</code>, dataset <code>0.1+5cc3c8d20c19</code>. Test split: 1,352 works whose artists, manuscripts/albums and near-duplicates do not occur in training.</em></p>
|
| <table>
|
| <thead>
|
| <tr>
|
| <th>Task</th>
|
| <th>Classes</th>
|
| <th>Logistic regression macro-F1 [95% CI]</th>
|
| <th>Balanced acc.</th>
|
| <th>k-NN retrieval macro-F1</th>
|
| <th>Majority macro-F1</th>
|
| </tr>
|
| </thead>
|
| <tbody>
|
| <tr>
|
| <td>region</td>
|
| <td>7</td>
|
| <td>0.705 [0.64–0.76]</td>
|
| <td>0.716</td>
|
| <td>0.658</td>
|
| <td>0.077</td>
|
| </tr>
|
| <tr>
|
| <td>period</td>
|
| <td>9</td>
|
| <td>0.420 [0.36–0.48]</td>
|
| <td>0.444</td>
|
| <td>0.397</td>
|
| <td>0.050</td>
|
| </tr>
|
| </tbody>
|
| </table>
|
| <p>Period predictions fall within one bin of the labelled bin for 78.2% of test works (mean distance to the documented date range 139 years).</p>
|
| <p>Collection-disjoint (train on one museum, test on the other): met → cleveland: region macro-F1 0.816 over 4 shared classes; cleveland → met: region macro-F1 0.662 over 5 shared classes.</p>
|
| <p>Full report: <code>artifacts/evaluation/runs/20261022T054811Z_e94d93bf34/report.md</code> and the Experiments page. These numbers describe these two collections only; see <code>docs/limitations.md</code>.</p>
|
|
|
|
|
| <h2>Contents</h2>
|
| <table>
|
| <thead>
|
| <tr>
|
| <th>Path</th>
|
| <th>What it is</th>
|
| </tr>
|
| </thead>
|
| <tbody>
|
| <tr>
|
| <td><code>src/artifex/</code></td>
|
| <td>the pipeline: ingestion, normalisation, labels, images, features, splits, training, retrieval, inference, evaluation</td>
|
| </tr>
|
| <tr>
|
| <td><code>scripts/</code></td>
|
| <td>command-line entry points (below)</td>
|
| </tr>
|
| <tr>
|
| <td><code>app/</code></td>
|
| <td>the Streamlit website (<code>app/main.py</code>, pages in <code>app/views/</code>)</td>
|
| </tr>
|
| <tr>
|
| <td><code>web/</code></td>
|
| <td>the in-browser edition (static HTML/JavaScript; the same model runs in the visitor's browser via ONNX Runtime Web)</td>
|
| </tr>
|
| <tr>
|
| <td><code>configs/</code></td>
|
| <td>every tunable choice: sources and filters, period bins, region taxonomy, preprocessing, classifiers, evaluation</td>
|
| </tr>
|
| <tr>
|
| <td><code>data/manifests/</code></td>
|
| <td>the versioned manifest, exclusions, region-mapping report, dataset statistics</td>
|
| </tr>
|
| <tr>
|
| <td><code>data/splits/</code></td>
|
| <td>split assignments</td>
|
| </tr>
|
| <tr>
|
| <td><code>artifacts/</code></td>
|
| <td>embeddings, trained heads, evaluation runs (generated)</td>
|
| </tr>
|
| <tr>
|
| <td><code>docs/</code></td>
|
| <td>research question, methodology, data sources, dataset card, taxonomy, evaluation protocol, limitations, deployment, roadmap</td>
|
| </tr>
|
| <tr>
|
| <td><code>tests/</code></td>
|
| <td>unit, integration and headless UI tests</td>
|
| </tr>
|
| </tbody>
|
| </table>
|
| <h2>Installation</h2>
|
| <p>Requires Python 3.10–3.12. A CUDA GPU is optional.</p>
|
| <pre><code class="language-bash">python -m venv .venv
|
| # Windows: .venv\Scripts\activate macOS/Linux: source .venv/bin/activate
|
| pip install -e ".[dev]"
|
| </code></pre>
|
| <p>For GPU acceleration install a CUDA build of PyTorch first, e.g.
|
| <code>pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124</code>.</p>
|
| <h2>Building everything (in order)</h2>
|
| <table>
|
| <thead>
|
| <tr>
|
| <th>Step</th>
|
| <th>Command</th>
|
| <th>Produces</th>
|
| <th>Time on the development machine</th>
|
| </tr>
|
| </thead>
|
| <tbody>
|
| <tr>
|
| <td>1. Ingest metadata</td>
|
| <td><code>python scripts/ingest_data.py</code></td>
|
| <td>raw API responses in <code>data/raw/</code></td>
|
| <td>Cleveland ≈ 2 min; the Met ≈ 3.5 h (its firewall limits clients to ≈ 1 request/s)</td>
|
| </tr>
|
| <tr>
|
| <td>2. Build dataset</td>
|
| <td><code>python scripts/build_dataset.py</code></td>
|
| <td>images in <code>data/processed/</code>, <code>data/manifests/*</code></td>
|
| <td>≈ 1 h of image downloads</td>
|
| </tr>
|
| <tr>
|
| <td>3. Embeddings</td>
|
| <td><code>python scripts/extract_embeddings.py</code> (add <code>--all-variants</code> for ablations)</td>
|
| <td><code>artifacts/embeddings/<variant>/</code></td>
|
| <td>≈ 2 min per DINOv2 variant for ~10,000 images on an RTX 3050 Laptop GPU (≈ 11 ms per image)</td>
|
| </tr>
|
| <tr>
|
| <td>4. Train</td>
|
| <td><code>python scripts/train_models.py</code></td>
|
| <td><code>data/splits/</code>, <code>artifacts/classifiers/deployed/</code></td>
|
| <td>≈ 1 min</td>
|
| </tr>
|
| <tr>
|
| <td>5. Evaluate</td>
|
| <td><code>python scripts/evaluate_models.py</code></td>
|
| <td><code>artifacts/evaluation/runs/<run>/</code>, <code>latest.json</code></td>
|
| <td>≈ 3.5 min with ablations (measured 200 s on a Ryzen 7 5800HS); see the run's <code>runtime_seconds</code></td>
|
| </tr>
|
| <tr>
|
| <td>6. Launch</td>
|
| <td><code>streamlit run app/main.py</code></td>
|
| <td><a href="http://localhost:8501">http://localhost:8501</a></td>
|
| <td></td>
|
| </tr>
|
| </tbody>
|
| </table>
|
| <p><strong>In-browser edition</strong> (static website, no server; see <a href="#/docs/deployment">docs/deployment.md</a>):
|
| <code>python scripts/export_browser_model.py</code>, then <code>python scripts/build_static_site.py</code>, <code>python scripts/validate_browser.py</code>
|
| (compares the browser pipeline with the Python application in headless Chromium) and <code>python scripts/build_static_site.py
|
| --no-thumbnails</code> to include that comparison; preview with <code>python scripts/serve_static.py</code>.</p>
|
| <p>All steps are resumable or idempotent: ingestion skips cached objects, image downloads skip existing files, embedding
|
| extraction reuses cached rows. Analyse one image from the command line with <code>python scripts/run_demo.py path/to/image.jpg</code>.</p>
|
| <h3>What is downloaded, generated, or not redistributed</h3>
|
| <table>
|
| <thead>
|
| <tr>
|
| <th>Resource</th>
|
| <th>How you get it</th>
|
| </tr>
|
| </thead>
|
| <tbody>
|
| <tr>
|
| <td>DINOv2 weights (<code>facebook/dinov2-base</code>, <code>-small</code>)</td>
|
| <td>downloaded automatically from the Hugging Face Hub on first use, pinned to a commit (<code>configs/preprocessing.yaml</code>) into <code>models/hf/</code></td>
|
| </tr>
|
| <tr>
|
| <td>Museum metadata and images</td>
|
| <td>fetched by steps 1–2 from the museums' official open-access APIs (CC0); not committed to Git</td>
|
| </tr>
|
| <tr>
|
| <td>Manifest, splits, statistics</td>
|
| <td>generated (small; can be committed for reproducibility)</td>
|
| </tr>
|
| <tr>
|
| <td>Embeddings, classifier heads, evaluation runs</td>
|
| <td>generated by steps 3–5</td>
|
| </tr>
|
| </tbody>
|
| </table>
|
| <p>The website shows explicit "dataset/model unavailable" and "no evaluation yet" states until the corresponding steps have
|
| run; it never shows placeholder predictions.</p>
|
| <h2>Using the website</h2>
|
| <ul>
|
| <li><strong>Home</strong> — what Artifex is, a sample of the collection, and the headline evaluation numbers of the latest run.</li>
|
| <li><strong>Analyse</strong> — upload a JPEG/PNG/WebP (≤ 20 MB). You get ranked region and period estimates with calibrated scores and
|
| abstention ("insufficient evidence"), a narrower tradition where supported, the five most similar reference works (with
|
| an explicit, reversible filter by predicted region or period), what the ten nearest works are documented as, and
|
| rule-based evidence statements (agreement/disagreement, training coverage, held-out performance for the predicted
|
| class, near-duplicate and out-of-distribution warnings). A JSON copy of the result can be downloaded.</li>
|
| <li><strong>Collection</strong> — search by title/artist/institution, filter by region, period, collection and medium, sort, page, and
|
| open a full record with provenance, licence, metadata caveats and visually closest works.</li>
|
| <li><strong>Methodology</strong> — the method (<code>docs/methodology.md</code>) and the live configuration: model version, encoder commit,
|
| classes, thresholds, bins, taxonomy and the full region-mapping report.</li>
|
| <li><strong>Experiments</strong> — every metric of the latest evaluation run: model comparisons with baselines, per-class tables,
|
| confusion matrices, calibration and risk–coverage curves, retrieval vs. random, collection-disjoint results, ablations,
|
| a gallery of confident mistakes and dataset composition; plus downloads of <code>metrics.json</code>, <code>predictions.csv</code>, <code>report.md</code>.</li>
|
| <li><strong>About</strong> — motivation, data and licence policy, privacy, stack, citation.</li>
|
| </ul>
|
| <p>Screenshots of the deployed in-browser edition (model <code>artifex-0.1-765e3b9a93</code>, headless Chromium at 1440 px and
|
| 390 px; the local Streamlit app has the same pages and content) are in <a href="docs/screenshots/desktop_analyse.jpg"><code>docs/screenshots/</code></a>: the
|
| analysis of a held-out test image (<a href="docs/screenshots/desktop_analyse.jpg">desktop</a>,
|
| <a href="docs/screenshots/mobile_analyse.jpg">mobile</a>), the <a href="docs/screenshots/desktop_experiments.jpg">experiments</a>,
|
| <a href="docs/screenshots/desktop_collection.jpg">collection</a>, <a href="docs/screenshots/desktop_collection_dialog.jpg">record</a>,
|
| <a href="docs/screenshots/desktop_home.jpg">home</a> and <a href="docs/screenshots/desktop_methodology.jpg">methodology</a> pages. The analysed
|
| image is itself in the reference collection, so it retrieves itself at similarity 1.00 and Artifex warns that the scores
|
| are not independent evidence. Artwork images in the screenshots come from the two museums' CC0 open-access releases.</p>
|
| <h2>Tests</h2>
|
| <pre><code class="language-bash">pytest # unit + integration + headless UI tests (offline, ~1–2 min)
|
| pytest -m slow # real DINOv2 encoder tests (download the checkpoint on first use)
|
| </code></pre>
|
| <p>The integration and UI tests run the real pipeline end to end on a small <strong>synthetic</strong> fixture (procedurally generated
|
| images with artificial labels) using the non-learned colour/texture encoder, so they need neither network access nor
|
| model downloads. The fixture exists only to test software; it is never presented as evidence.</p>
|
| <h2>Troubleshooting</h2>
|
| <table>
|
| <thead>
|
| <tr>
|
| <th>Symptom</th>
|
| <th>Cause / fix</th>
|
| </tr>
|
| </thead>
|
| <tbody>
|
| <tr>
|
| <td>Met ingestion logs <code>throttled (403)</code></td>
|
| <td>The Met's firewall rate-limits bursts. The client backs off automatically; lower <code>sources.met.http.requests_per_second</code> if it persists. Re-running resumes.</td>
|
| </tr>
|
| <tr>
|
| <td>"The dataset changed after the model was trained"</td>
|
| <td>The manifest was rebuilt. Run steps 3–5 again.</td>
|
| </tr>
|
| <tr>
|
| <td>"Cached embeddings do not match the trained model"</td>
|
| <td>Preprocessing or encoder config changed. Re-run steps 3–5.</td>
|
| </tr>
|
| <tr>
|
| <td>Analysis page says model or dataset unavailable</td>
|
| <td>One of steps 1–4 has not been run; the page lists which.</td>
|
| </tr>
|
| <tr>
|
| <td>Slow first analysis</td>
|
| <td>The encoder loads once per app process (≈ 10 s).</td>
|
| </tr>
|
| <tr>
|
| <td>No GPU used</td>
|
| <td>Install a CUDA build of PyTorch; <code>--device cpu</code> forces CPU.</td>
|
| </tr>
|
| <tr>
|
| <td>Art Institute of Chicago</td>
|
| <td>Not used: its API/IIIF server blocked programmatic clients during development (see <code>docs/data_sources.md</code>).</td>
|
| </tr>
|
| </tbody>
|
| </table>
|
| <h2>Documentation</h2>
|
| <p><a href="#/docs/research_question">Research question</a> · <a href="#/docs/methodology">Methodology</a> ·
|
| <a href="#/docs/data_sources">Data sources</a> · <a href="#/docs/dataset_card">Dataset card</a> · <a href="#/docs/label_taxonomy">Label taxonomy</a> ·
|
| <a href="#/docs/annotation_protocol">Annotation protocol</a> · <a href="#/docs/evaluation_protocol">Evaluation protocol</a> ·
|
| <a href="#/docs/limitations">Limitations</a> · <a href="#/docs/deployment">Deployment</a> · <a href="#/docs/roadmap">Roadmap</a></p>
|
| <h2>Licence and acknowledgements</h2>
|
| <p>Code: MIT (<code>LICENSE</code>). Reference data and images: The Metropolitan Museum of Art Open Access and The Cleveland Museum of
|
| Art Open Access, CC0 1.0. DINOv2: Meta AI, Apache-2.0. See <code>CITATION.cff</code> for citation.</p> |
