File size: 3,485 Bytes
a23acdf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | <p>Artifex v0.1 uses <strong>no manual labels</strong>. Every label is derived deterministically from museum metadata by code and
configuration that can be reviewed. This document describes how those derivations are reviewed and how manual review
should be done if it is added.</p>
<h2>Automatic labelling (current)</h2>
<table>
<thead>
<tr>
<th>Label</th>
<th>Source fields</th>
<th>Rule</th>
<th>Review artefact</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>date_start</code>, <code>date_end</code></td>
<td>Met <code>objectBeginDate</code>/<code>objectEndDate</code>; Cleveland <code>creation_date_earliest</code>/<code>latest</code></td>
<td>historical → astronomical years; unusable values recorded in <code>date_issue</code></td>
<td><code>manifest.csv</code> (<code>original_date_text</code> beside the range)</td>
</tr>
<tr>
<td><code>period_label</code></td>
<td>date range</td>
<td>≥ 75 % of range inside one bin</td>
<td><code>period_status</code> column</td>
</tr>
<tr>
<td><code>region_normalized</code>, <code>culture_normalized</code></td>
<td>culture, place fields, artist nationality, Egyptian Art department</td>
<td>ordered regex rules; multi-region matches → ambiguous</td>
<td><code>region_mapping_report.csv</code></td>
</tr>
<tr>
<td>eligibility</td>
<td>licence flags, classification, object name, title</td>
<td>see <code>configs/dataset.yaml</code> → <code>filters</code></td>
<td><code>exclusions.csv</code></td>
</tr>
<tr>
<td>duplicates</td>
<td>image URL, image bytes</td>
<td>first artwork id is canonical</td>
<td><code>duplicate_of</code> column</td>
</tr>
</tbody>
</table>
<h2>Reviewing the region mapping</h2>
<ol>
<li>Open <code>data/manifests/region_mapping_report.csv</code> (one row per distinct source string per field, with counts).</li>
<li>Sort by <code>count</code>. Check every <code>unmapped</code> and <code>ambiguous</code> string with ≥ 3 records, and a sample of <code>assigned</code> strings
per broad region.</li>
<li>If a mapping is wrong, change <code>configs/region_mapping.yaml</code> (never the manifest), bump its <code>version</code>, rebuild:
<code>python scripts/build_dataset.py</code>. The dataset version changes, which invalidates the trained model and evaluation,
forcing re-training and re-evaluation.</li>
<li>Record the reason for each rule change in the commit message.</li>
</ol>
<p>Decisions taken so far (see <code>label_taxonomy.md</code>): ambiguous strings stay unlabelled; explicit "Painting: X" clauses win;
compound terms such as "Roman Egypt" use override rules; dynasty names are not used as place indicators except where
the dynasty is unambiguous about place (Safavid, Qajar).</p>
<h2>If manual annotation is added</h2>
<ul>
<li>Annotators label from the <strong>museum record and scholarly sources</strong>, never from the model's prediction (the prediction
must be hidden during annotation to avoid anchoring).</li>
<li>Each manual label stores: annotator id, date, source consulted, confidence (certain / probable / uncertain), and free-text
justification. Manual labels live in a separate file keyed by <code>artwork_id</code>; the original museum fields are never edited.</li>
<li>At least 10 % of manually labelled records are double-annotated; agreement (Cohen's κ) is reported.</li>
<li>Manual corrections are evaluated as a separate label set, so results with and without them can be compared.</li>
</ul> |