Model card for 263702

README.md

@@ -11,174 +11,126 @@ tags:
 
 # clockface
 
-Reads an analog clock and returns the time. Finds the dial first, then reads it.
-11.5 MB of weights.
+Reads an analog clock and returns the time. Finds the dial with a
+COCO-pretrained detector, then reads the crop.
 
-> **It does not work on real photographs.** Off by 155.9 minutes on average
-> across 1,728 of them. Guessing at random is off by 180. On the synthetic
-> renders it trained on it is off by 4.8 minutes. I am publishing it because
-> that gap is the only interesting thing here.
+> Best measured result on 200 held-out real photographs: **34.0 minutes**
+> mean error, **1.2** median, **49.0%** within one minute.
+> Guessing at random is off by 180 minutes. It is better than it was and it is
+> not a working clock reader.
 
-Two weight sets are here. `stage2_best.pt` was trained on renders alone.
-`v3mix_stage2.pt` was trained on rougher renders plus 1,528 real photographs,
-to see whether real data closed the gap. It did not, and both are published
-rather than only the flattering one.
+Version 263702, commit `efe5c96`. Reader backbone: `resnet50`.
 
-| weights | version | commit |
-|---|---|---|
-| `stage2_best.pt` + `stage1_best.pt` | 263601 | `82d3030` |
-| `v3mix_stage2.pt` | 263701 | `deed0ca` |
+## Results on real photographs
 
-Both were briefly published under the same version number. The release counter
-is derived from git tags, and with no tags it returns 01 forever, so two
-different models were stamped 263701. Tags now exist for both, and the stamp
-says UNTAGGED when none do rather than emitting a number that looks unique and
-is not.
+200 held-out photos, third-party labels, all read through the same pipeline:
+COCO detector finds the dial, the reader reads the crop.
 
-## The metric
+| reader | MAE | median | within 1 min | within 5 min | worse than 30 min |
+|---|---|---|---|---|---|
+| `resnet50` | 34.0 min | 1.2 | 49.0% | 73.0% | 22.0% |
+| `resnet18` | 48.5 min | 1.5 | 44.5% | 60.5% | 34.5% |
+| `mobilenet_v3_small` | 55.6 min | 2.5 | 34.5% | 59.0% | 36.5% |
+| random guessing | 180 min | | | | |
 
-A clock face is a ring of 720 minutes, and error is the shorter way round it. So
-11:58 against 12:02 is 4 minutes, not 716. AM and PM do not exist, because a face
-cannot show them: 3:47 and 15:47 are the same reading.
+## What stage 1 is, and what it is not
 
-Random guessing scores 180 minutes. That is the bar.
+Stage 1 is `fasterrcnn_mobilenet_v3_large_fpn` with COCO weights, run on CPU. It
+is not trained here and not fine-tuned. Three ways of producing a crop, same
+reader, same 200 photographs:
 
-## What it scores on real photographs
-
-1,728 images with third-party labels.
-
-| | MAE | median | within 5 min | worse than 30 min |
-|---|---|---|---|---|
-| two stages | 155.9 min | 139.9 | 11.4% | 82.2% |
-| whole image, no locator | 151.8 min | 133.7 | 9.2% | 82.5% |
-| random guessing | 180 min | | | |
-
-Dropping the locator makes it slightly better, which tells you how much the
-locator is contributing.
-
-The images are 1,092 film stills from Marclay's *The Clock*, 506 photos from
-`moondream/analog-clock-benchmark`, and 131 cropped faces from
-`oliverj990/clock-faces-v1-times`. Most of those labels I have not checked. On
-the 106 I did check by hand against the image, the error is 148.5 minutes, so bad
-labels are not the excuse.
-
-Split by source, the story is obvious:
-
-| source | MAE | within 5 min |
-|---|---|---|
-| cropped faces, dial fills the frame | 87.8 min | 45.0% |
-| photographs of rooms and streets | 163.3 min | 9.3% |
-
-The closer a photo looks to my renders, the better it does. That is a domain gap
-and nothing subtler.
-
-## What it scores on synthetic data
-
-1,200 held-out renders.
-
-| | MAE | median | within 1 min | within 5 min |
-|---|---|---|---|---|
-| ground-truth crops | 4.82 min | 0.42 | 79.0% | 92.5% |
-| end to end | 5.07 min | 0.43 | 79.0% | 92.2% |
-
-Do not quote these. The renderer that made the labels also made the pixels, so
-the model is being marked by its own teacher. They are here for one reason: 4.8
-minutes on renders against 156 on photographs is the whole story.
-
-## Does training on real photographs help? No.
-
-`v3mix_stage2.pt` was trained on a rougher synthetic corpus (sensor noise up to
-sigma 42, JPEG down to q25, occlusion, far wider hand geometry) plus 1,528 real
-photographs, holding out 200 for evaluation. 106 of those 200 have labels I
-checked by hand against the image; the rest are third-party and unverified.
-
-On the 106 labels I actually verified:
-
-| weights | trained on | MAE | within 5 min |
+| crop | MAE | median | within 5 min |
 |---|---|---|---|
-| stage2_best.pt | renders only | 148.5 min | 10.4% |
-| v3mix_stage2.pt | rougher renders + 1,528 real photos | 151.9 min | 12.3% |
-| random guessing | | 180 min | |
-
-Adding real photographs moved it from 148.5 to 151.9, which is to say nowhere.
+| whole image, no locator | 120.6 min | 88.3 | 22.5% |
+| a dial locator trained on renders (2.2 MB) | 120.7 min | 67.9 | 25.0% |
+| COCO detector (78 MB) | 73.6 min | 18.4 | 45.5% |
 
-On the full 200-photo held-out set v3mix scores MAE 120.6 with 24.5% within
-five minutes, which looks better and mostly is not: that set contains the
-pre-cropped clock faces, which are the easy ones. Its own verified subset scores
-151.9. Composition, not skill.
+The locator trained on synthetic renders contributes nothing over having no
+locator at all. It was removed. Distilling the detector into it would not have
+helped: a student cannot beat its teacher, and the training would have run back
+through the renderer whose domain gap is the thing being escaped.
 
-Its synthetic score is 35.0 minutes against the older model's 4.8, because the
-renders it trained on are much harder. That is the intended direction and it
-still did not transfer.
+Detection runs on CPU because sustained torchvision detection on an M2 trips the
+GPU watchdog. On CPU it is 0.135s per image and finds a clock in 93-96% of real
+photographs, having never seen a render.
 
-## Where it goes wrong
+## How the two hands become one time
 
-Of the 1,421 predictions off by more than half an hour, 720 would improve if you
-swapped the hour and minute hands. My generator drew hands from a narrow band of
-lengths and widths, and the model learned that band instead of learning which
-hand is longer. Real clocks do not agree with my band.
+A clock's hands are geared: at time t the hour hand sits at t/720 of a turn and
+the minute hand at (t mod 60)/60 of a turn, so one number fixes both. The reader
+predicts a distribution for each hand, and the decoder searches the 720 minutes
+the clock could be showing for the one that best explains both. It cannot return
+a pair of angles that describes no time, and a confident minute hand can pull a
+vague hour hand onto the right hour.
 
-The confidence signal does not survive the trip to real data. On renders it works
-well: keep the most confident 75% and error drops from 4.82 to 1.64 minutes. On
-photographs it does nothing, 155.9 to 153.5. The model is confidently wrong,
-which is worse than being uncertain.
-
-The locator finds a dial to within 1.5% of image width on renders. On a photo of
-a room where the clock is a smudge on a far wall, it does not.
-
-## How it is built
-
-    stage 1   DialLocator        554K params, 2.2 MB    image -> centre, radius
-    stage 2   PretrainedReader   2.3M params, 9.2 MB    crop -> two angle distributions
-
-Stage 2 is MobileNetV3-Small pretrained on ImageNet, with two heads over 180
-angular bins each. Targets are von Mises bumps rather than one-hot, so being one
-bin out costs less than being ten out, and decoding uses a circular soft-argmax
-to get back below bin resolution.
-
-The same network without ImageNet weights never learns this at all. It sits at
-2·ln(180), a flat distribution, for as long as you care to train it, while
-happily memorising 400 images. The pretrained initialisation is the difference
-between a model and a plateau.
-
-It predicts both hands and never the time directly. That costs nothing and buys
-two things. The hands disagreeing is a confidence signal. And decoding can work
-like a vernier scale, taking precision from the minute hand and only the hour
-count from the hour hand, which under noise is about 11 times more accurate than
-reading the hour hand alone.
-
-## Using it
-
-    import torch
-    from twostage import PretrainedReader
-    from model import decode_cls
-
-    reader = PretrainedReader(bins=180, arch="mobilenet_v3_small")
-    reader.load_state_dict(torch.load("stage2_best.pt")["model"])
-    hour_logits, minute_logits = reader(crop)      # 1x3x256x256, ImageNet normalised
-    minutes, hour_only, disagreement, sharpness = decode_cls(hour_logits, minute_logits)
-
-`minutes` is the position on the 720-minute ring. `disagreement` is how far apart
-the two hands' readings are, in minutes.
-
-## What it is for
-
-Round analog clock faces. Not digital-analog hybrids, subdials, chronographs or
-24-hour faces.
-
-Do not use it for anything where a wrong time costs something. It is off by more
-than half an hour on 82% of real photographs.
-
-## Reproducing
-
-    git clone https://github.com/lyte-codes/clockface
-    ./eval.py --self-test
-    python3 predict.py --labels <labels> --root <dir> \
-        --stage2 stage2_best.pt --stage1 stage1_best.pt --out preds.jsonl
-    ./eval.py --labels <labels> --pred preds.jsonl
-
-`--self-test` runs 32 checks on the metric itself, including both directions
-across the 12 o'clock seam.
+| decoder | MAE | median | within 5 min | worse than 30 min |
+|---|---|---|---|---|
+| each hand read on its own | 35.3 min | 1.4 | 69.0% | 25.5% |
+| search over times the clock could show | 34.0 min | 1.2 | 73.0% | 22.0% |
+
+Same weights, same crops, same photographs; only the decoding differs. The gain
+is in readings that were nearly right already, which is why MAE barely moves --
+MAE here is set by the readings that are wrong by hours, and those are wrong in
+the image, not in the decoding.
+
+Scoring the hands in each other's roles, to catch the ones read the wrong way
+round, does not work and is off by default. On these photographs the exchanged
+reading scored a median 0.06 better where the hands really were swapped and 0.11
+worse everywhere else -- the two populations sit on top of each other. A swap is
+not two good angles in the wrong slots; the model is confidently wrong about
+both hands at once and they agree on a wrong but perfectly geared time.
+
+## When to trust a reading
+
+The reader ranks its own answers, and the ranking is worth more than the
+headline. Keeping only the half it is surest of:
+
+```
+200 readings. Keeping only the ones each signal is most sure of:
+
+  hand agreement
+       keep     n      MAE   median   <=5min
+       100%   200    34.02     1.25    73.0%
+        75%   150    26.80     1.00    79.3%
+        50%   100    22.64     0.75    86.0%
+        25%    50    30.12     0.75    82.0%
+        10%    20    20.74     0.50    90.0%
+    rank correlation with error +0.352 (useful)
+
+  head sharpness
+       keep     n      MAE   median   <=5min
+       100%   200    34.02     1.25    73.0%
+        75%   150    16.58     0.75    84.0%
+        50%   100     8.18     0.50    94.0%
+        25%    50     1.91     0.50    98.0%
+        10%    20     4.03     0.50    95.0%
+    rank correlation with error -0.506 (useful)
+
+  search margin
+       keep     n      MAE   median   <=5min
+       100%   200    34.02     1.25    73.0%
+        75%   150    18.39     0.75    84.0%
+        50%   100    10.73     0.50    92.0%
+        25%    50     6.75     0.50    96.0%
+        10%    20     3.75     0.50    95.0%
+    rank correlation with error -0.496 (useful)
+
+  A caution on reading the low-coverage rows: at 25% of 200 photographs a row
+  rests on 50 readings, so small differences between signals there are noise.
+```
+
+The signal this project set out to use for that was the disagreement between the
+hands, which is free to compute. It works on renders and barely ranks anything
+on photographs. The two that do work came out of the decoder instead.
+
+## Honest limits
+
+The labels on these 200 photographs are third party. 106 of them were checked by
+hand against the image; the rest were not. The photographs were collected for
+other purposes and skew toward clocks that are already the subject of the frame.
+
+There is no first-party test set. A number measured on somebody else's labels is
+worth less than one measured on your own, and that gap is not closed here.
 
 ## Licence