Model card: measured results, including the sim-to-real failure

README.md

@@ -1,3 +1,132 @@
 ---
 license: apache-2.0
+datasets:
+- lyte-codes/clockface-synth-dataset-10k
+pipeline_tag: image-to-text
+tags:
+- analog-clock
+- time-reading
+- sim2real
 ---
+
+# clockface
+
+Reads an analog clock face and returns the time. Two stages: find the dial,
+then read it. 11.5 MB of weights in total.
+
+> **This model does not work well on real photographs.** On 1,728 real photos it
+> scores **155.9 minutes** mean error, against **180 minutes for guessing at
+> random**. It scores 4.8 minutes on the synthetic data it was trained on. It is
+> published because the gap between those two numbers is the useful part.
+
+Version 263701. Trained on synth v2, revision `main` of the dataset above.
+
+## The metric
+
+A clock face is a ring of 720 minutes. Error is the shorter way round it, so it
+lies in [0, 360] and 11:58 against 12:02 is 4 minutes, not 716. AM/PM does not
+exist: a face cannot show it, so 3:47 and 15:47 are the same reading.
+
+Guessing uniformly at random scores 180 minutes. That is the number to beat, and
+it is the number this model is measured against.
+
+## Results
+
+### Real photographs (1,728 images, third-party labels)
+
+| | MAE | median | within 5 min | worse than 30 min |
+|---|---|---|---|---|
+| two-stage | 155.9 min | 139.9 | 11.4% | 82.2% |
+| whole image, no locator | 151.8 min | 133.7 | 9.2% | 82.5% |
+| **random guessing** | **180 min** | — | — | — |
+
+Sources: 1,092 film stills from Marclay's *The Clock*, 506 photographs from
+`moondream/analog-clock-benchmark`, 131 cropped faces from
+`oliverj990/clock-faces-v1-times`. Their labels are third party and mostly
+unverified; restricted to the 106 labels checked by hand against the image, the
+result is MAE 148.5, so label noise does not explain it.
+
+By source, the pattern is a domain gap and not much else:
+
+| source | MAE | within 5 min |
+|---|---|---|
+| cropped clock faces (dial fills frame) | 87.8 min | 45.0% |
+| photographs of scenes | 163.3 min | 9.3% |
+
+The closer an image is to the synthetic framing, the better it does.
+
+### Synthetic validation (1,200 held-out renders) — training telemetry only
+
+| | MAE | median | within 1 min | within 5 min |
+|---|---|---|---|---|
+| ground-truth crops | 4.82 min | 0.42 | 79.0% | 92.5% |
+| end to end | 5.07 min | 0.43 | 79.0% | 92.2% |
+
+**These numbers are not a result.** The model is being scored by the same
+renderer that produced its labels. They are here to show the size of the
+sim-to-real gap, which is the point: 4.8 minutes on renders, 156 on photographs.
+
+## What goes wrong
+
+- **Hand confusion.** 720 of the 1,421 gross failures would improve if the hour
+  and minute hands were swapped. The generator used a narrow band of hand
+  lengths and widths, and the model learned that convention rather than learning
+  which hand is longer.
+- **The confidence signal does not survive.** The two hands' disagreement is a
+  good confidence signal on synthetic data (MAE 4.82 falls to 1.64 when keeping
+  the most confident 75%) and is inert on real photographs (155.9 to 153.5). The
+  model is confidently wrong, which is worse than being uncertain.
+- **Small clocks.** The locator finds a dial to 1.5% of image width on renders
+  and fails on wide scenes where the clock is a small part of the frame.
+
+## Architecture
+
+    stage 1  DialLocator      554K params, 2.2 MB   image -> dial centre, radius
+    stage 2  PretrainedReader 2.3M params, 9.2 MB   crop -> two angle distributions
+
+Stage 2 is a MobileNetV3-Small backbone pretrained on ImageNet, with two heads
+predicting a distribution over 180 angular bins for each hand. Targets are von
+Mises bumps rather than one-hot, and decoding takes a circular soft-argmax.
+
+An identical from-scratch network never leaves chance on this task: it sits at
+2*ln(180), a uniform distribution, however long it trains, while overfitting 400
+images perfectly. ImageNet initialisation is what makes it learn at all.
+
+Both hands are predicted, never the time directly, so their disagreement gives a
+confidence signal for free. Decoding is a vernier: the minute hand supplies
+precision, the hour hand picks which hour. Measured against reading the hour hand
+alone, that is about 11x more accurate under noise.
+
+## Usage
+
+    import torch
+    from twostage import DialLocator, PretrainedReader, crop_dial
+    from model import decode_cls
+
+    reader = PretrainedReader(bins=180, arch="mobilenet_v3_small")
+    reader.load_state_dict(torch.load("stage2_best.pt")["model"])
+    hour_logits, minute_logits = reader(crop)          # crop: 1x3x256x256, ImageNet norm
+    minutes, hour_only, disagreement, sharpness = decode_cls(hour_logits, minute_logits)
+
+`minutes` is the position on the 720-minute ring; `disagreement` is how far apart
+the two hands' readings are, in minutes.
+
+## Intended use
+
+Reading round analog clock faces. Out of scope: digital-analog hybrids, subdials,
+chronographs, 24-hour faces.
+
+**Not suitable for any use where a wrong time carries a cost.** It is wrong by
+more than half an hour on 82% of real photographs.
+
+## Reproducing
+
+    git clone https://github.com/lyte-codes/clockface
+    ./eval.py --self-test                              # 32/32, checks the metric
+    python3 predict.py --labels <labels> --root <dir> \
+        --stage2 stage2_best.pt --stage1 stage1_best.pt --out preds.jsonl
+    ./eval.py --labels <labels> --pred preds.jsonl
+
+## Licence
+
+Apache 2.0. The synthetic dataset is CC BY 4.0.