onatm commited on
Commit
ba812a5
·
verified ·
1 Parent(s): 621d868

Update model card

Browse files
Files changed (1) hide show
  1. README.md +7 -1
README.md CHANGED
@@ -49,7 +49,11 @@ The response includes `questions.route.answer`, `questions.route.probabilities`
49
 
50
  ## Evaluation
51
 
52
- Clean-question scores from the saved reports. Accuracy is at raw T=1 (temperature scaling does not change the winning answer); Brier and ECE are lower-is-better. The served columns use the saved temperature **T=1.6245**. A dash means calibrated metrics were not recorded for that split.
 
 
 
 
53
 
54
  | Suite / split | Clean n | Accuracy | Brier raw | Brier served | ECE raw | ECE served |
55
  | --- | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -58,6 +62,8 @@ Clean-question scores from the saved reports. Accuracy is at raw T=1 (temperatur
58
  | decision-v7/test | 1,200 | 0.8292 | 0.2457 | 0.2336 | 0.0661 | 0.0170 |
59
  | transfer-v4/test | 656 | 0.6250 | 0.4680 | 0.4491 | 0.1136 | 0.0571 |
60
 
 
 
61
  `decision-v7` contains held-out questions from the training source families; `transfer-v4` contains new sources and held-out policy structures. Development was for model selection; these reports describe seed 0. Temperature was fitted on the separate `decision-v7/calibration` split, not on either test split.
62
 
63
  The decision-v7 test report predates the temperature fit. Its served metrics were computed afterward from saved raw logits, without rerunning inference or fitting on test.
 
49
 
50
  ## Evaluation
51
 
52
+ Clean-question scores from the saved reports. Accuracy is at raw T=1 (temperature scaling does not change the winning answer); Brier and ECE are lower-is-better. A dash means calibrated metrics were not recorded for that split.
53
+
54
+ <!-- gev:eval -->
55
+
56
+ The served columns use the saved temperature **T=1.6245**.
57
 
58
  | Suite / split | Clean n | Accuracy | Brier raw | Brier served | ECE raw | ECE served |
59
  | --- | ---: | ---: | ---: | ---: | ---: | ---: |
 
62
  | decision-v7/test | 1,200 | 0.8292 | 0.2457 | 0.2336 | 0.0661 | 0.0170 |
63
  | transfer-v4/test | 656 | 0.6250 | 0.4680 | 0.4491 | 0.1136 | 0.0571 |
64
 
65
+ <!-- /gev:eval -->
66
+
67
  `decision-v7` contains held-out questions from the training source families; `transfer-v4` contains new sources and held-out policy structures. Development was for model selection; these reports describe seed 0. Temperature was fitted on the separate `decision-v7/calibration` split, not on either test split.
68
 
69
  The decision-v7 test report predates the temperature fit. Its served metrics were computed afterward from saved raw logits, without rerunning inference or fitting on test.