Model card for 263703

README.md

@@ -14,123 +14,109 @@ tags:
 Reads an analog clock and returns the time. Finds the dial with a
 COCO-pretrained detector, then reads the crop.
 
-> Best measured result on 200 held-out real photographs: **34.0 minutes**
-> mean error, **1.2** median, **49.0%** within one minute.
-> Guessing at random is off by 180 minutes. It is better than it was and it is
-> not a working clock reader.
+> **263703 reads 51.5% of 200 held-out real photographs to the minute**,
+> against 49.0% for 263702. It is also wrong by more than half an hour
+> more often -- 25.5% against 22.0% -- so this is a
+> trade, not a clean win. Which one you want depends on whether a wrong answer
+> costs you more than a missing one. Both are in this repository.
 
-Version 263702, commit `efe5c96`. Reader backbone: `resnet50`.
+Version 263703, commit `1fce71d`. Reader backbone: `resnet50`.
 
-## Results on real photographs
+## The two versions, same 200 photographs, same pipeline
 
-200 held-out photos, third-party labels, all read through the same pipeline:
-COCO detector finds the dial, the reader reads the crop.
-
-| reader | MAE | median | within 1 min | within 5 min | worse than 30 min |
+| | to the minute | median | within 5 min | MAE | worse than 30 min |
 |---|---|---|---|---|---|
-| `resnet50` | 34.0 min | 1.2 | 49.0% | 73.0% | 22.0% |
-| `resnet18` | 48.5 min | 1.5 | 44.5% | 60.5% | 34.5% |
-| `mobilenet_v3_small` | 55.6 min | 2.5 | 34.5% | 59.0% | 36.5% |
-| random guessing | 180 min | | | | |
-
-## What stage 1 is, and what it is not
-
-Stage 1 is `fasterrcnn_mobilenet_v3_large_fpn` with COCO weights, run on CPU. It
-is not trained here and not fine-tuned. Three ways of producing a crop, same
-reader, same 200 photographs:
-
-| crop | MAE | median | within 5 min |
-|---|---|---|---|
-| whole image, no locator | 120.6 min | 88.3 | 22.5% |
-| a dial locator trained on renders (2.2 MB) | 120.7 min | 67.9 | 25.0% |
-| COCO detector (78 MB) | 73.6 min | 18.4 | 45.5% |
-
-The locator trained on synthetic renders contributes nothing over having no
-locator at all. It was removed. Distilling the detector into it would not have
-helped: a student cannot beat its teacher, and the training would have run back
-through the renderer whose domain gap is the thing being escaped.
-
-Detection runs on CPU because sustained torchvision detection on an M2 trips the
-GPU watchdog. On CPU it is 0.135s per image and finds a clock in 93-96% of real
-photographs, having never seen a render.
-
-## How the two hands become one time
-
-A clock's hands are geared: at time t the hour hand sits at t/720 of a turn and
-the minute hand at (t mod 60)/60 of a turn, so one number fixes both. The reader
-predicts a distribution for each hand, and the decoder searches the 720 minutes
-the clock could be showing for the one that best explains both. It cannot return
-a pair of angles that describes no time, and a confident minute hand can pull a
-vague hour hand onto the right hour.
-
-| decoder | MAE | median | within 5 min | worse than 30 min |
-|---|---|---|---|---|
-| each hand read on its own | 35.3 min | 1.4 | 69.0% | 25.5% |
-| search over times the clock could show | 34.0 min | 1.2 | 73.0% | 22.0% |
-
-Same weights, same crops, same photographs; only the decoding differs. The gain
-is in readings that were nearly right already, which is why MAE barely moves --
-MAE here is set by the readings that are wrong by hours, and those are wrong in
-the image, not in the decoding.
-
-Scoring the hands in each other's roles, to catch the ones read the wrong way
-round, does not work and is off by default. On these photographs the exchanged
-reading scored a median 0.06 better where the hands really were swapped and 0.11
-worse everywhere else -- the two populations sit on top of each other. A swap is
-not two good angles in the wrong slots; the model is confidently wrong about
-both hands at once and they agree on a wrong but perfectly geared time.
+| **263703** | **51.5%** | **1.00 min** | 69.0% | 40.4 min | 25.5% |
+| 263702 | 49.0% | 1.25 min | **73.0%** | **34.0 min** | **22.0%** |
+| guessing | ~0% | 180 min | 3% | 180 min | 97% |
+
+The trade has a cause, and it is visible while training. Measured on real
+photographs held out from training, epoch by epoch: readings correct to the
+minute climb from 23.2% to 48.7% across all 22 epochs and never stop climbing,
+while mean error bottoms out at epoch 8 and then gets worse. Training longer
+makes the model sharper on the clocks it can read and less steady on the ones it
+cannot. There is no epoch that is best at both, so the choice is a choice.
+
+## What changed
+
+**The reader gained a third head.** The other two each describe one hand, so a
+dial whose hands are hard to tell apart corrupts both at once -- on 263702, 27.5%
+of readings got the minute right and the hour wrong. The third head is asked for
+the time itself over the whole 12-hour ring, coarsely, with no notion of a hand:
+it cannot express a role confusion because it never assigns roles, and it only
+has to break the twelve-way tie. In a controlled comparison -- same data, same
+recipe, only the head differing -- it moved exactly what it was aimed at, hours
+missed with the minute correct falling from 12.0% to 8.5%.
+
+**The checkpoint is chosen on photographs, not renders.** Renders stop
+predicting real accuracy about a third of the way through training; picking the
+epoch renders liked cost 3 to 7 minutes of real error. It is chosen on
+photographs held out from training and never in this test set, and by the
+to-the-minute rate, because MAE and the to-the-minute rate disagree about which
+epoch is best and only one of them is what this model is for.
 
 ## When to trust a reading
 
 The reader ranks its own answers, and the ranking is worth more than the
-headline. Keeping only the half it is surest of:
+headline. Declining the quarter it is least sure of takes it from
+69.0% within five minutes to 96%.
 
 ```
 200 readings. Keeping only the ones each signal is most sure of:
 
   hand agreement
        keep     n      MAE   median   <=5min
-       100%   200    34.02     1.25    73.0%
-        75%   150    26.80     1.00    79.3%
-        50%   100    22.64     0.75    86.0%
-        25%    50    30.12     0.75    82.0%
-        10%    20    20.74     0.50    90.0%
-    rank correlation with error +0.352 (useful)
+       100%   200    40.40     1.00    69.0%
+        75%   150    26.06     1.00    80.0%
+        50%   100    15.16     0.75    86.0%
+        25%    50    10.31     0.50    90.0%
+        10%    20     7.20     0.75    95.0%
+    rank correlation with error +0.388 (useful)
 
   head sharpness
        keep     n      MAE   median   <=5min
-       100%   200    34.02     1.25    73.0%
-        75%   150    16.58     0.75    84.0%
-        50%   100     8.18     0.50    94.0%
-        25%    50     1.91     0.50    98.0%
-        10%    20     4.03     0.50    95.0%
-    rank correlation with error -0.506 (useful)
+       100%   200    40.40     1.00    69.0%
+        75%   150    24.20     0.75    80.7%
+        50%   100     7.80     0.75    92.0%
+        25%    50     1.84     0.75    98.0%
+        10%    20     3.55     0.50    95.0%
+    rank correlation with error -0.504 (useful)
 
   search margin
        keep     n      MAE   median   <=5min
-       100%   200    34.02     1.25    73.0%
-        75%   150    18.39     0.75    84.0%
-        50%   100    10.73     0.50    92.0%
-        25%    50     6.75     0.50    96.0%
-        10%    20     3.75     0.50    95.0%
-    rank correlation with error -0.496 (useful)
+       100%   200    40.40     1.00    69.0%
+        75%   150    21.12     0.75    82.0%
+        50%   100    10.33     0.75    91.0%
+        25%    50     5.75     0.50    96.0%
+        10%    20     0.57     0.50   100.0%
+    rank correlation with error -0.529 (useful)
 
   A caution on reading the low-coverage rows: at 25% of 200 photographs a row
   rests on 50 readings, so small differences between signals there are noise.
 ```
 
-The signal this project set out to use for that was the disagreement between the
-hands, which is free to compute. It works on renders and barely ranks anything
-on photographs. The two that do work came out of the decoder instead.
+The signal this project set out to use for that was disagreement between the
+hands. On weaker readers it ranked almost nothing; it becomes useful here, which
+took a good enough model rather than a better signal.
+
+## Stage 1
+
+`fasterrcnn_mobilenet_v3_large_fpn` with COCO weights, on CPU, not fine-tuned.
+It finds a clock in 93-96% of real photographs having never seen a render. A
+dial locator trained on renders scored no better than not cropping at all and
+was removed. On the 9 test photographs where stage 1 finds nothing, 1 is still
+read within five minutes.
 
 ## Honest limits
 
-The labels on these 200 photographs are third party. 106 of them were checked by
-hand against the image; the rest were not. The photographs were collected for
-other purposes and skew toward clocks that are already the subject of the frame.
+The goal this was pointed at was reading the time to the minute 95% of the time.
+It does not get there: 51.5% across all photographs, and about 90% even
+when allowed to decline 85% of them.
 
-There is no first-party test set. A number measured on somebody else's labels is
-worth less than one measured on your own, and that gap is not closed here.
+The labels on these 200 photographs are third party. 106 were checked by hand
+against the image; the rest were not. There is no first-party test set, and a
+number measured on somebody else's labels is worth less than one measured on
+your own.
 
 ## Licence