Spaces:

SuckSome
/

artifex

Running

App Files Files Community

artifex / data /docs /evaluation_protocol.html

SuckSome

Claude Opus 5.5

Deploy model artifex-0.1-e94d93bf34 (artist grouping by painter roles) with browser validation

db295a6 verified 11 days ago

Raw History Blame Contribute Delete

9.52 kB

<p>Configuration: <code>configs/evaluation.yaml</code>. Implementation: <code>src/artifex/splits.py</code>, <code>src/artifex/evaluate.py</code>,
<code>src/artifex/metrics.py</code>, <code>src/artifex/uncertainty.py</code>.</p>
<h2>1. Leakage control</h2>
<p>Closely related works must never be split between training and test. Records are connected (union-find) when they share:</p>
<table>
<thead>
<tr>
<th>Link</th>
<th>Rationale</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Artist</strong> — normalised name (accents, punctuation and parenthetical qualifiers removed), across both museums; only creators recorded in a painter role (artist, painter, illuminator, draftsman, workshop director)</td>
<td>Prevents learning an artist's hand rather than a tradition. Followers/workshops recorded under the master's name are grouped with the master. Authors of a manuscript's text, calligraphers, poets and patrons are not used: an earlier build linked every page whose text was by, e.g., Firdausi, chaining 712 pages of unrelated Persian and Indian manuscripts into one group, of which the per-group cap then kept 25.</td>
</tr>
<tr>
<td><strong>Accession lot</strong> (e.g. <code>1962.279.*</code>) for anonymous works, and for any record whose title marks it as a folio, leaf, page or album/series part</td>
<td>Folios of one manuscript or leaves of one album share hands, materials and photography.</td>
</tr>
<tr>
<td><strong>Near-duplicate image</strong> — DINOv2 cosine ≥ 0.95</td>
<td>Catches different records showing the same or a nearly identical image (details, replicas, re-photographs).</td>
</tr>
</tbody>
</table>
<p>Exact duplicates (same image URL or identical bytes) are removed earlier, at dataset build time.</p>
<p><strong>Group cap.</strong> At most 25 works per group (chosen deterministically by hashing the artwork id with the seed) enter the
<em>modelling subset</em> used for training and evaluation; the rest remain in the collection and the retrieval index. Without
the cap a single 600-folio manuscript would dominate its class.</p>
<p>The split code asserts that no group spans two splits, and the retrieval evaluation reports the share of retrieved
training works from the query's own group (<code>same_group@k</code>, expected 0).</p>
<h2>2. Splits</h2>
<ul>
<li><strong>Main (artist-disjoint)</strong>: <code>StratifiedGroupKFold</code> with 20 folds (fixed seed), stratified on broad region; 14 folds
  train, 3 validation, 3 test (≈70/15/15). Used for model selection (validation) and the reported results (test).</li>
<li><strong>Collection-disjoint</strong>: for each ordered pair of museums, train (and validate, 85/15 grouped) on one museum and test
  on the other. Test works whose group (e.g. artist) also occurs in the training museum are removed. Only classes with
  ≥ 30 training works and ≥ 10 test works are evaluated; if fewer than two classes qualify, the experiment is reported as
  not evaluable rather than run on a degenerate label set.</li>
</ul>
<p>Split assignments (<code>data/splits/splits.parquet</code>, <code>.csv</code>), the seed and the configuration hash are saved; each evaluation
run copies them into its run directory.</p>
<p><strong>Supported classes.</strong> A label becomes a classifier target when the modelling subset contains at least
<code>min_records_per_class</code> (60) works with that label (40 for narrower traditions). This is a dataset-design decision made
from label counts only — no model output is involved — and it is applied identically to every model compared. Test works
whose label is not supported are reported separately (section 4).</p>
<h2>3. Models compared</h2>
<table>
<thead>
<tr>
<th>Model</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>Majority class</td>
<td>Predicts the most frequent training class.</td>
</tr>
<tr>
<td>Logistic regression (deployed)</td>
<td>Multinomial, balanced class weights, <code>C</code> chosen on validation macro-F1.</td>
</tr>
<tr>
<td>Linear SVM</td>
<td>Same standardised features, balanced class weights, <code>C ∈ {0.0003, 0.001, 0.003}</code> chosen on validation macro-F1 (no probabilities; classification metrics only). Main split only — it is a comparison baseline and liblinear is slow on 1,536-dimensional features.</td>
</tr>
<tr>
<td>k-NN retrieval vote</td>
<td>Similarity-weighted vote of the 10 nearest <em>training</em> works. Represents "retrieval only".</td>
</tr>
<tr>
<td>Logistic regression + k-NN</td>
<td>Mean of calibrated classifier scores and k-NN vote shares ("combined presentation").</td>
</tr>
</tbody>
</table>
<h2>4. Metrics</h2>
<h3>Region (and narrower traditions)</h3>
<p>Macro-F1 (primary — classes are imbalanced), balanced accuracy, accuracy, per-class precision/recall/F1 with support,
confusion matrix. Narrower-tradition heads are evaluated on test works whose documented broad region is the parent
("oracle" broad region).</p>
<h3>Period</h3>
<p>All of the above on bin labels, plus:
- <strong>Within-tolerance accuracy</strong>: predicted bin within ±1 position of the labelled bin (bins in chronological order).
- <strong>Range-compatible accuracy</strong>: the predicted bin overlaps the documented date range at all.
- <strong>Absolute error (years)</strong>: distance from the predicted bin's midpoint to the documented range, zero if inside; mean and
  median reported.
- <strong>Imprecisely dated works</strong>: test works whose range spans several bins (no label) are scored with range-compatible
  accuracy and absolute error only.</p>
<p>These are complementary: exact-bin accuracy penalises a 1549/1550 boundary miss as heavily as a 500-year error; error in
years does not.</p>
<h3>Calibration and selective prediction</h3>
<p>Expected calibration error (10 equal-width bins on the top-class score) and negative log-likelihood, before and after
temperature scaling; reliability diagrams; risk–coverage curves (selective accuracy as a function of coverage); test
coverage and selective accuracy at the validation-chosen abstention threshold.</p>
<h3>Unsupported categories</h3>
<p>Test works whose label is not a supported class are reported separately: their count, the abstention rate, and what the
head predicted for them. This is a partial out-of-scope probe; no dedicated out-of-distribution benchmark (e.g.
photographs, sculpture) is included in v0.1.</p>
<h3>Retrieval</h3>
<p>Test works are queries; training works are the reference set. Relevance uses documented metadata only:
- <code>region_hit@k</code> — at least one of the top k shares the query's broad region;
- <code>region_precision@k</code>, <code>period_precision@k</code> — share of the top k with the same label (unlabelled never matches);
- <code>temporal_close@k</code> — share of the top k whose documented range lies within 50 years of the query's range;
- <code>top1_median_gap_years</code>;
- the same metrics computed on <strong>random rankings</strong> give the chance level.</p>
<p>Retrieval is not judged only by label identity: two works can be visually and historically related across regions.
These metrics measure agreement with catalogue metadata, nothing more.</p>
<h3>Uncertainty of the metrics</h3>
<p>For the main experiment, test-split macro-F1, balanced accuracy and accuracy of every model carry 95% percentile
intervals from 1,000 bootstrap resamples of <strong>whole leakage groups</strong> (works within a group are correlated, so resampling
individual works would understate the uncertainty).</p>
<h3>Classifier–retrieval agreement</h3>
<p>Agreement rate between the deployed classifier and the k-NN vote, and accuracy when they agree vs. disagree.</p>
<h2>5. Ablations</h2>
<p>Same works, same split, same classifiers; only the representation changes:</p>
<table>
<thead>
<tr>
<th>Variant</th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td>DINOv2-B, letterbox, colour (default)</td>
<td>reference</td>
</tr>
<tr>
<td>DINOv2-B, centre crop</td>
<td>Does cropping away borders/format change performance?</td>
</tr>
<tr>
<td>DINOv2-B, greyscale</td>
<td>How much depends on colour (palette, varnish, paper tone)?</td>
</tr>
<tr>
<td>DINOv2-S</td>
<td>Does a smaller encoder suffice?</td>
</tr>
<tr>
<td>Colour/texture statistics</td>
<td>A non-learned low-level baseline (HSV histogram, gradient-orientation histograms, global statistics).</td>
</tr>
</tbody>
</table>
<p>These comparisons are diagnostic, not causal explanations of model behaviour.</p>
<h2>6. Artefacts</h2>
<p><code>artifacts/evaluation/runs/&lt;UTC timestamp&gt;_&lt;model hash&gt;/</code>:
<code>metrics.json</code> (all numbers), <code>predictions.csv</code> (per-example predictions for validation and test),
<code>failures.json</code> (confident mistakes), <code>report.md</code> (generated from <code>metrics.json</code>), <code>configs/</code> (snapshot), <code>splits.csv</code>,
<code>model_card.json</code>. <code>artifacts/evaluation/latest.json</code> points to the newest run; the website reads only these files.</p>
<h2>7. What is <em>not</em> claimed</h2>
<p>No single number summarises the model. Results apply to the two source collections, their photography and their
cataloguing conventions. Calibration and abstention thresholds are validated on data from the same distribution and do
not transfer automatically to arbitrary user photographs.</p>

	<p>Configuration: <code>configs/evaluation.yaml</code>. Implementation: <code>src/artifex/splits.py</code>, <code>src/artifex/evaluate.py</code>,
	<code>src/artifex/metrics.py</code>, <code>src/artifex/uncertainty.py</code>.</p>
	<h2>1. Leakage control</h2>
	<p>Closely related works must never be split between training and test. Records are connected (union-find) when they share:</p>
	<table>
	<thead>
	<tr>
	<th>Link</th>
	<th>Rationale</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td><strong>Artist</strong> — normalised name (accents, punctuation and parenthetical qualifiers removed), across both museums; only creators recorded in a painter role (artist, painter, illuminator, draftsman, workshop director)</td>
	<td>Prevents learning an artist's hand rather than a tradition. Followers/workshops recorded under the master's name are grouped with the master. Authors of a manuscript's text, calligraphers, poets and patrons are not used: an earlier build linked every page whose text was by, e.g., Firdausi, chaining 712 pages of unrelated Persian and Indian manuscripts into one group, of which the per-group cap then kept 25.</td>
	</tr>
	<tr>
	<td><strong>Accession lot</strong> (e.g. <code>1962.279.*</code>) for anonymous works, and for any record whose title marks it as a folio, leaf, page or album/series part</td>
	<td>Folios of one manuscript or leaves of one album share hands, materials and photography.</td>
	</tr>
	<tr>
	<td><strong>Near-duplicate image</strong> — DINOv2 cosine ≥ 0.95</td>
	<td>Catches different records showing the same or a nearly identical image (details, replicas, re-photographs).</td>
	</tr>
	</tbody>
	</table>
	<p>Exact duplicates (same image URL or identical bytes) are removed earlier, at dataset build time.</p>
	<p><strong>Group cap.</strong> At most 25 works per group (chosen deterministically by hashing the artwork id with the seed) enter the
	<em>modelling subset</em> used for training and evaluation; the rest remain in the collection and the retrieval index. Without
	the cap a single 600-folio manuscript would dominate its class.</p>
	<p>The split code asserts that no group spans two splits, and the retrieval evaluation reports the share of retrieved
	training works from the query's own group (<code>same_group@k</code>, expected 0).</p>
	<h2>2. Splits</h2>
	<ul>
	<li><strong>Main (artist-disjoint)</strong>: <code>StratifiedGroupKFold</code> with 20 folds (fixed seed), stratified on broad region; 14 folds
	train, 3 validation, 3 test (≈70/15/15). Used for model selection (validation) and the reported results (test).</li>
	<li><strong>Collection-disjoint</strong>: for each ordered pair of museums, train (and validate, 85/15 grouped) on one museum and test
	on the other. Test works whose group (e.g. artist) also occurs in the training museum are removed. Only classes with
	≥ 30 training works and ≥ 10 test works are evaluated; if fewer than two classes qualify, the experiment is reported as
	not evaluable rather than run on a degenerate label set.</li>
	</ul>
	<p>Split assignments (<code>data/splits/splits.parquet</code>, <code>.csv</code>), the seed and the configuration hash are saved; each evaluation
	run copies them into its run directory.</p>
	<p><strong>Supported classes.</strong> A label becomes a classifier target when the modelling subset contains at least
	<code>min_records_per_class</code> (60) works with that label (40 for narrower traditions). This is a dataset-design decision made
	from label counts only — no model output is involved — and it is applied identically to every model compared. Test works
	whose label is not supported are reported separately (section 4).</p>
	<h2>3. Models compared</h2>
	<table>
	<thead>
	<tr>
	<th>Model</th>
	<th>Description</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td>Majority class</td>
	<td>Predicts the most frequent training class.</td>
	</tr>
	<tr>
	<td>Logistic regression (deployed)</td>
	<td>Multinomial, balanced class weights, <code>C</code> chosen on validation macro-F1.</td>
	</tr>
	<tr>
	<td>Linear SVM</td>
	<td>Same standardised features, balanced class weights, <code>C ∈ {0.0003, 0.001, 0.003}</code> chosen on validation macro-F1 (no probabilities; classification metrics only). Main split only — it is a comparison baseline and liblinear is slow on 1,536-dimensional features.</td>
	</tr>
	<tr>
	<td>k-NN retrieval vote</td>
	<td>Similarity-weighted vote of the 10 nearest <em>training</em> works. Represents "retrieval only".</td>
	</tr>
	<tr>
	<td>Logistic regression + k-NN</td>
	<td>Mean of calibrated classifier scores and k-NN vote shares ("combined presentation").</td>
	</tr>
	</tbody>
	</table>
	<h2>4. Metrics</h2>
	<h3>Region (and narrower traditions)</h3>
	<p>Macro-F1 (primary — classes are imbalanced), balanced accuracy, accuracy, per-class precision/recall/F1 with support,
	confusion matrix. Narrower-tradition heads are evaluated on test works whose documented broad region is the parent
	("oracle" broad region).</p>
	<h3>Period</h3>
	<p>All of the above on bin labels, plus:
	- <strong>Within-tolerance accuracy</strong>: predicted bin within ±1 position of the labelled bin (bins in chronological order).
	- <strong>Range-compatible accuracy</strong>: the predicted bin overlaps the documented date range at all.
	- <strong>Absolute error (years)</strong>: distance from the predicted bin's midpoint to the documented range, zero if inside; mean and
	median reported.
	- <strong>Imprecisely dated works</strong>: test works whose range spans several bins (no label) are scored with range-compatible
	accuracy and absolute error only.</p>
	<p>These are complementary: exact-bin accuracy penalises a 1549/1550 boundary miss as heavily as a 500-year error; error in
	years does not.</p>
	<h3>Calibration and selective prediction</h3>
	<p>Expected calibration error (10 equal-width bins on the top-class score) and negative log-likelihood, before and after
	temperature scaling; reliability diagrams; risk–coverage curves (selective accuracy as a function of coverage); test
	coverage and selective accuracy at the validation-chosen abstention threshold.</p>
	<h3>Unsupported categories</h3>
	<p>Test works whose label is not a supported class are reported separately: their count, the abstention rate, and what the
	head predicted for them. This is a partial out-of-scope probe; no dedicated out-of-distribution benchmark (e.g.
	photographs, sculpture) is included in v0.1.</p>
	<h3>Retrieval</h3>
	<p>Test works are queries; training works are the reference set. Relevance uses documented metadata only:
	- <code>region_hit@k</code> — at least one of the top k shares the query's broad region;
	- <code>region_precision@k</code>, <code>period_precision@k</code> — share of the top k with the same label (unlabelled never matches);
	- <code>temporal_close@k</code> — share of the top k whose documented range lies within 50 years of the query's range;
	- <code>top1_median_gap_years</code>;
	- the same metrics computed on <strong>random rankings</strong> give the chance level.</p>
	<p>Retrieval is not judged only by label identity: two works can be visually and historically related across regions.
	These metrics measure agreement with catalogue metadata, nothing more.</p>
	<h3>Uncertainty of the metrics</h3>
	<p>For the main experiment, test-split macro-F1, balanced accuracy and accuracy of every model carry 95% percentile
	intervals from 1,000 bootstrap resamples of <strong>whole leakage groups</strong> (works within a group are correlated, so resampling
	individual works would understate the uncertainty).</p>
	<h3>Classifier–retrieval agreement</h3>
	<p>Agreement rate between the deployed classifier and the k-NN vote, and accuracy when they agree vs. disagree.</p>
	<h2>5. Ablations</h2>
	<p>Same works, same split, same classifiers; only the representation changes:</p>
	<table>
	<thead>
	<tr>
	<th>Variant</th>
	<th>Purpose</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td>DINOv2-B, letterbox, colour (default)</td>
	<td>reference</td>
	</tr>
	<tr>
	<td>DINOv2-B, centre crop</td>
	<td>Does cropping away borders/format change performance?</td>
	</tr>
	<tr>
	<td>DINOv2-B, greyscale</td>
	<td>How much depends on colour (palette, varnish, paper tone)?</td>
	</tr>
	<tr>
	<td>DINOv2-S</td>
	<td>Does a smaller encoder suffice?</td>
	</tr>
	<tr>
	<td>Colour/texture statistics</td>
	<td>A non-learned low-level baseline (HSV histogram, gradient-orientation histograms, global statistics).</td>
	</tr>
	</tbody>
	</table>
	<p>These comparisons are diagnostic, not causal explanations of model behaviour.</p>
	<h2>6. Artefacts</h2>
	<p><code>artifacts/evaluation/runs/<UTC timestamp>_<model hash>/</code>:
	<code>metrics.json</code> (all numbers), <code>predictions.csv</code> (per-example predictions for validation and test),
	<code>failures.json</code> (confident mistakes), <code>report.md</code> (generated from <code>metrics.json</code>), <code>configs/</code> (snapshot), <code>splits.csv</code>,
	<code>model_card.json</code>. <code>artifacts/evaluation/latest.json</code> points to the newest run; the website reads only these files.</p>
	<h2>7. What is <em>not</em> claimed</h2>
	<p>No single number summarises the model. Results apply to the two source collections, their photography and their
	cataloguing conventions. Calibration and abstention thresholds are validated on data from the same distribution and do
	not transfer automatically to arbitrary user photographs.</p>