Multiple Choice
PEFT
Safetensors
English
gev
decision-model
lora
pointer-head
calibration
Eval Results (legacy)
Instructions to use onatm/gev-e2b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use onatm/gev-e2b with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Update model card
Browse files
README.md
CHANGED
|
@@ -49,7 +49,11 @@ The response includes `questions.route.answer`, `questions.route.probabilities`
|
|
| 49 |
|
| 50 |
## Evaluation
|
| 51 |
|
| 52 |
-
Clean-question scores from the saved reports. Accuracy is at raw T=1 (temperature scaling does not change the winning answer); Brier and ECE are lower-is-better.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
|
| 54 |
| Suite / split | Clean n | Accuracy | Brier raw | Brier served | ECE raw | ECE served |
|
| 55 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
@@ -58,6 +62,8 @@ Clean-question scores from the saved reports. Accuracy is at raw T=1 (temperatur
|
|
| 58 |
| decision-v7/test | 1,200 | 0.8292 | 0.2457 | 0.2336 | 0.0661 | 0.0170 |
|
| 59 |
| transfer-v4/test | 656 | 0.6250 | 0.4680 | 0.4491 | 0.1136 | 0.0571 |
|
| 60 |
|
|
|
|
|
|
|
| 61 |
`decision-v7` contains held-out questions from the training source families; `transfer-v4` contains new sources and held-out policy structures. Development was for model selection; these reports describe seed 0. Temperature was fitted on the separate `decision-v7/calibration` split, not on either test split.
|
| 62 |
|
| 63 |
The decision-v7 test report predates the temperature fit. Its served metrics were computed afterward from saved raw logits, without rerunning inference or fitting on test.
|
|
|
|
| 49 |
|
| 50 |
## Evaluation
|
| 51 |
|
| 52 |
+
Clean-question scores from the saved reports. Accuracy is at raw T=1 (temperature scaling does not change the winning answer); Brier and ECE are lower-is-better. A dash means calibrated metrics were not recorded for that split.
|
| 53 |
+
|
| 54 |
+
<!-- gev:eval -->
|
| 55 |
+
|
| 56 |
+
The served columns use the saved temperature **T=1.6245**.
|
| 57 |
|
| 58 |
| Suite / split | Clean n | Accuracy | Brier raw | Brier served | ECE raw | ECE served |
|
| 59 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
|
|
| 62 |
| decision-v7/test | 1,200 | 0.8292 | 0.2457 | 0.2336 | 0.0661 | 0.0170 |
|
| 63 |
| transfer-v4/test | 656 | 0.6250 | 0.4680 | 0.4491 | 0.1136 | 0.0571 |
|
| 64 |
|
| 65 |
+
<!-- /gev:eval -->
|
| 66 |
+
|
| 67 |
`decision-v7` contains held-out questions from the training source families; `transfer-v4` contains new sources and held-out policy structures. Development was for model selection; these reports describe seed 0. Temperature was fitted on the separate `decision-v7/calibration` split, not on either test split.
|
| 68 |
|
| 69 |
The decision-v7 test report predates the temperature fit. Its served metrics were computed afterward from saved raw logits, without rerunning inference or fitting on test.
|