SeanLiu commited on
Commit
dc2d645
·
verified ·
1 Parent(s): 9b77fa5

card: correct the calibration note (AUROC definitions), add the frozen-backbone comparison on long text

Browse files
Files changed (1) hide show
  1. README.md +5 -4
README.md CHANGED
@@ -47,10 +47,11 @@ general item, 150 to 190 ms per screenshot with all of a step's questions, on on
47
 
48
  ## Known limits
49
 
50
- - Out of its training domains it is over-confident: on JevBench's public hard tier (text-only long documents) it is
51
- 52% accurate at 91% mean confidence. Yes/no ranking on general images (AUROC 0.84 to 0.91) is below the frozen
52
- backbone's (0.96). A temperature fitted on out-of-domain data is the intended fix; V10 tests a readout that keeps
53
- the backbone's yes/no geometry.
 
54
  - Web grounding on Mind2Web-style pages 0.823 and macOS desktop grounding 0.822 are below the earlier grounding-only
55
  checkpoint (0.850 / 0.914).
56
  - Trained and evaluated on 1280x1000 desktop web screenshots; not on mobile, logged-in or form-heavy flows.
 
47
 
48
  ## Known limits
49
 
50
+ - On text-only long documents (JevBench's public hard tier) it is 52% accurate at 91% mean confidence: badly
51
+ over-confident out of domain. The frozen backbone read from next-token logits is 59% at ECE 0.35 on the same items,
52
+ so most of this is the backbone's own behaviour on long text; a temperature fitted on out-of-domain images
53
+ (T = 1.3 on GQA val) moves the hard-tier ECE from 0.28 to 0.24 only. Do not use the stated probabilities on long
54
+ text documents as calibrated confidences. On general images the yes/no class AUROC is 0.97 to 0.98.
55
  - Web grounding on Mind2Web-style pages 0.823 and macOS desktop grounding 0.822 are below the earlier grounding-only
56
  checkpoint (0.850 / 0.914).
57
  - Trained and evaluated on 1280x1000 desktop web screenshots; not on mobile, logged-in or form-heavy flows.