card: correct the calibration note (AUROC definitions), add the frozen-backbone comparison on long text
Browse files
README.md
CHANGED
|
@@ -47,10 +47,11 @@ general item, 150 to 190 ms per screenshot with all of a step's questions, on on
|
|
| 47 |
|
| 48 |
## Known limits
|
| 49 |
|
| 50 |
-
-
|
| 51 |
-
|
| 52 |
-
backbone's
|
| 53 |
-
the
|
|
|
|
| 54 |
- Web grounding on Mind2Web-style pages 0.823 and macOS desktop grounding 0.822 are below the earlier grounding-only
|
| 55 |
checkpoint (0.850 / 0.914).
|
| 56 |
- Trained and evaluated on 1280x1000 desktop web screenshots; not on mobile, logged-in or form-heavy flows.
|
|
|
|
| 47 |
|
| 48 |
## Known limits
|
| 49 |
|
| 50 |
+
- On text-only long documents (JevBench's public hard tier) it is 52% accurate at 91% mean confidence: badly
|
| 51 |
+
over-confident out of domain. The frozen backbone read from next-token logits is 59% at ECE 0.35 on the same items,
|
| 52 |
+
so most of this is the backbone's own behaviour on long text; a temperature fitted on out-of-domain images
|
| 53 |
+
(T = 1.3 on GQA val) moves the hard-tier ECE from 0.28 to 0.24 only. Do not use the stated probabilities on long
|
| 54 |
+
text documents as calibrated confidences. On general images the yes/no class AUROC is 0.97 to 0.98.
|
| 55 |
- Web grounding on Mind2Web-style pages 0.823 and macOS desktop grounding 0.822 are below the earlier grounding-only
|
| 56 |
checkpoint (0.850 / 0.914).
|
| 57 |
- Trained and evaluated on 1280x1000 desktop web screenshots; not on mobile, logged-in or form-heavy flows.
|