Hugging Face's logo

electroglyph
/
Spark-X2.6-4B-benchmarks-gpqa

Model card Files
xet
 Community
electroglyph commited on
Commit
5df47cc
·
verified ·
1 Parent(s): e31bb7f

Add calibrated PIQA column and correlations

Browse files
Files changed (1) hide show
  1. README.md +87 -58
README.md CHANGED
@@ -2,12 +2,17 @@
2
 
3
  Which predicts GPQA better across 50 KLD-stratified Heretic trials: **PIQA** or **low KL divergence**?
4
 
5
- **Answer: inconclusive.** Neither predictor beats the other.
 
 
6
 
7
- Each of the 50 model variants has three scores: PIQA (commonsense quiz),
8
- KL (how much safety tuning changed it), GPQA (hard science quiz). Rank the
 
9
  variants by PIQA and by KL; whichever ranking matches the GPQA ranking
10
- better is the better predictor.
 
 
11
 
12
  How the test was run:
13
 
@@ -31,6 +36,14 @@ seed, graded by option letter, 32-row waves on one 95 GiB GPU.
31
  6. Win rule, fixed beforehand: lead by 0.10 or more with fluke odds
32
  under 1 in 20. KL led by 0.163 but its fluke odds were 1 in 10. No
33
  winner.
 
 
 
 
 
 
 
 
34
 
35
  What came out: the PIQA ranking barely matches GPQA; chance alone
36
  matches it that well 6 times in 10. The KL ranking matches better, but
@@ -38,6 +51,19 @@ chance matches it that well 1 time in 10. The bar is 1 in 20. Neither
38
  clears it. KL looks better, but with 50 variants that gap could be
39
  luck. More variants or bigger gaps would settle it.
40
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
  ## Result (n=50)
42
 
43
  Same result as numbers. Rank agreement: -1 opposite order, 0 no
@@ -48,6 +74,7 @@ two other ways and agree.
48
  | --- | --- | --- | --- | --- | --- |
49
  | PIQA vs GPQA | +0.069 | 0.629 (about 6 in 10) | +0.042 | 0.668 | +0.240 |
50
  | low-KLD vs GPQA | +0.232 | 0.104 (about 1 in 10) | +0.160 | 0.101 | +0.280 |
 
51
 
52
  ## Method
53
 
@@ -55,69 +82,71 @@ two other ways and agree.
55
  - Candidates: 50 trials stratified across KL 0.003–1.666, refusals <= 8/100, KL <= 2.0.
56
  - PIQA: full validation set, `acc_norm`, batch `auto` (same batching as the journal scores for comparability); unmodified-model baseline PIQA = 0.6763.
57
  - GPQA: first 50 usable rows of GPQA-Diamond (`Idavidrein/gpqa`, 6 multi-line rows skipped), options permuted per record (`permute-v1`), graded by option letter; 1 run, temp 0.6, top_p 0.95, seed 20261024, max 16384 new tokens.
58
- - Harness: `bench_spark2.py`; generated 2026-10-28T04:01:10Z; 50/50 benchmarked, 0 failed.
 
59
 
60
- GPQA range across candidates: 0.02–0.56 (mean 0.40). PIQA range: 0.663–0.683 (mean 0.674).
61
 
62
  ## Candidates
63
 
64
- | trial | KL | refusals/100 | PIQA | GPQA |
65
- | --- | --- | --- | --- | --- |
66
- | 799 | 0.0027 | 6.0 | 0.6746 | 0.50 |
67
- | 94 | 0.0277 | 5.0 | 0.6763 | 0.44 |
68
- | 89 | 0.0289 | 4.0 | 0.6774 | 0.38 |
69
- | 177 | 0.0298 | 5.0 | 0.6703 | 0.40 |
70
- | 209 | 0.0301 | 3.0 | 0.6768 | 0.26 |
71
- | 179 | 0.0302 | 3.0 | 0.6719 | 0.42 |
72
- | 178 | 0.0309 | 6.0 | 0.6741 | 0.36 |
73
- | 175 | 0.0311 | 6.0 | 0.6687 | 0.50 |
74
- | 167 | 0.0321 | 4.0 | 0.6736 | 0.44 |
75
- | 985 | 0.0330 | 4.0 | 0.6746 | 0.44 |
76
- | 155 | 0.0362 | 2.0 | 0.6741 | 0.56 |
77
- | 189 | 0.0365 | 4.0 | 0.6774 | 0.48 |
78
- | 152 | 0.0366 | 5.0 | 0.6725 | 0.42 |
79
- | 996 | 0.0369 | 0.0 | 0.6763 | 0.54 |
80
- | 33 | 0.0387 | 5.0 | 0.6817 | 0.46 |
81
- | 160 | 0.0393 | 3.0 | 0.6741 | 0.38 |
82
- | 184 | 0.0395 | 3.0 | 0.6763 | 0.52 |
83
- | 504 | 0.0409 | 7.0 | 0.6725 | 0.46 |
84
- | 903 | 0.0450 | 0.0 | 0.6757 | 0.36 |
85
- | 861 | 0.0483 | 5.0 | 0.6774 | 0.48 |
86
- | 158 | 0.0485 | 3.0 | 0.6752 | 0.40 |
87
- | 1026 | 0.0497 | 2.0 | 0.6768 | 0.30 |
88
- | 899 | 0.0503 | 1.0 | 0.6768 | 0.32 |
89
- | 64 | 0.0529 | 4.0 | 0.6790 | 0.40 |
90
- | 945 | 0.0530 | 0.0 | 0.6741 | 0.36 |
91
- | 953 | 0.0562 | 5.0 | 0.6746 | 0.46 |
92
- | 50 | 0.0594 | 5.0 | 0.6719 | 0.44 |
93
- | 892 | 0.0877 | 8.0 | 0.6708 | 0.52 |
94
- | 63 | 0.0924 | 3.0 | 0.6785 | 0.32 |
95
- | 58 | 0.1013 | 6.0 | 0.6746 | 0.52 |
96
- | 434 | 0.1020 | 6.0 | 0.6757 | 0.42 |
97
- | 882 | 0.1138 | 0.0 | 0.6708 | 0.20 |
98
- | 376 | 0.1138 | 6.0 | 0.6719 | 0.48 |
99
- | 13 | 0.1259 | 7.0 | 0.6757 | 0.50 |
100
- | 1099 | 0.1283 | 5.0 | 0.6746 | 0.54 |
101
- | 881 | 0.1354 | 1.0 | 0.6632 | 0.14 |
102
- | 34 | 0.1477 | 4.0 | 0.6714 | 0.38 |
103
- | 1054 | 0.2197 | 0.0 | 0.6676 | 0.02 |
104
- | 16 | 0.2380 | 6.0 | 0.6692 | 0.48 |
105
- | 1077 | 0.2443 | 1.0 | 0.6703 | 0.42 |
106
- | 907 | 0.2916 | 4.0 | 0.6708 | 0.48 |
107
- | 42 | 0.3233 | 5.0 | 0.6736 | 0.38 |
108
- | 931 | 0.3271 | 5.0 | 0.6736 | 0.48 |
109
- | 15 | 0.3674 | 2.0 | 0.6817 | 0.46 |
110
- | 1078 | 0.3926 | 1.0 | 0.6834 | 0.28 |
111
- | 691 | 0.3939 | 3.0 | 0.6714 | 0.38 |
112
- | 36 | 0.4163 | 4.0 | 0.6697 | 0.42 |
113
- | 71 | 0.7175 | 4.0 | 0.6687 | 0.24 |
114
- | 1086 | 0.8243 | 0.0 | 0.6757 | 0.26 |
115
- | 23 | 1.6658 | 0.0 | 0.6719 | 0.32 |
116
 
117
  ## Files
118
 
119
  - `benchmarks/results.json` - full records, correlations, config
120
  - `benchmarks/piqa_results.jsonl`, `benchmarks/gpqa_results.jsonl`
 
121
  - `benchmarks/partial/*.jsonl` - per-item generation checkpoints
122
  - `benchmarks/gpqa_vs_piqa.png`, `benchmarks/gpqa_vs_kld.png`
123
  - `logs/` - run logs, faults/watchdog dumps, crash reports
 
2
 
3
  Which predicts GPQA better across 50 KLD-stratified Heretic trials: **PIQA** or **low KL divergence**?
4
 
5
+ **Answer: inconclusive.** Neither predictor beats the other. Calibrated
6
+ PIQA ranks GPQA at +0.179 (fluke odds about 2 in 10): better than plain
7
+ PIQA, worse than low KL, still nowhere near the bar.
8
 
9
+ Each of the 50 model variants has four scores: PIQA (commonsense quiz),
10
+ calibrated PIQA (commonsense quiz, uncertain questions only), KL (how much
11
+ safety tuning changed it), GPQA (hard science quiz). Rank the
12
  variants by PIQA and by KL; whichever ranking matches the GPQA ranking
13
+ better is the better predictor. A later follow-up scored calibrated PIQA
14
+ on the same 50 and checked it the same way; it lands between the two and
15
+ changes nothing about the answer.
16
 
17
  How the test was run:
18
 
 
36
  6. Win rule, fixed beforehand: lead by 0.10 or more with fluke odds
37
  under 1 in 20. KL led by 0.163 but its fluke odds were 1 in 10. No
38
  winner.
39
+ 7. Calibrated follow-up: re-scored the same 50 adapters on the base
40
+ model's uncertain PIQA questions only. 1,195 of 1,838 validation
41
+ questions came out split across 3 samples on the base model; each
42
+ candidate answered those 5 times (5,975 answers) at the model's own
43
+ temperature (1.0), top_p 0.95. Base model scored 3,254/5,975. Same
44
+ adapters, same replay check as step 3. This comparison was designed
45
+ after the GPQA numbers were in, so its fluke odds are description,
46
+ not a registered test.
47
 
48
  What came out: the PIQA ranking barely matches GPQA; chance alone
49
  matches it that well 6 times in 10. The KL ranking matches better, but
 
51
  clears it. KL looks better, but with 50 variants that gap could be
52
  luck. More variants or bigger gaps would settle it.
53
 
54
+ Calibrated PIQA (post-hoc, so the fluke odds below describe rather than
55
+ test): ranking GPQA by the flip-set score agrees at +0.179, fluke odds
56
+ about 2 in 10 -- a touch better than plain PIQA's +0.069, worse than low
57
+ KL's +0.232, still far from the bar. The surprise is what the calibrated
58
+ score does track: low KL at +0.757, fluke odds about 1 in 10 million. It
59
+ barely tracks plain PIQA at all (+0.104) or refusals (+0.258). So the
60
+ flip-set score behaves like a finer damage meter -- it sees the same
61
+ thing KL divergence sees, with wider spread across candidates
62
+ (0.503-0.580 vs PIQA's 0.663-0.683) -- not like a better GPQA predictor.
63
+ Which makes sense in hindsight: the flip set is defined by the base
64
+ model's uncertainty, and decensoring moves candidates off the base
65
+ model's answers.
66
+
67
  ## Result (n=50)
68
 
69
  Same result as numbers. Rank agreement: -1 opposite order, 0 no
 
74
  | --- | --- | --- | --- | --- | --- |
75
  | PIQA vs GPQA | +0.069 | 0.629 (about 6 in 10) | +0.042 | 0.668 | +0.240 |
76
  | low-KLD vs GPQA | +0.232 | 0.104 (about 1 in 10) | +0.160 | 0.101 | +0.280 |
77
+ | calibrated-PIQA vs GPQA | +0.179 | 0.211 (about 2 in 10) | +0.126 | 0.197 | +0.309 |
78
 
79
  ## Method
80
 
 
82
  - Candidates: 50 trials stratified across KL 0.003–1.666, refusals <= 8/100, KL <= 2.0.
83
  - PIQA: full validation set, `acc_norm`, batch `auto` (same batching as the journal scores for comparability); unmodified-model baseline PIQA = 0.6763.
84
  - GPQA: first 50 usable rows of GPQA-Diamond (`Idavidrein/gpqa`, 6 multi-line rows skipped), options permuted per record (`permute-v1`), graded by option letter; 1 run, temp 0.6, top_p 0.95, seed 20261024, max 16384 new tokens.
85
+ - Calibrated PIQA: the 1,195 validation questions the base model split on across 3 samples at its own temperature (1.0); 5 samples per candidate (base 3254/5975 = 0.5446). Harness: `bench_spark3.py`, scorer `heretic.scorers.calibrated_piqa` @ `electroglyph/heretic` `master` (`369715e` at scoring time); merged 2026-10-28T10:36:57Z.
86
+ - Harness: `bench_spark2.py`; generated 2026-10-28T04:01:10Z; 50/50 benchmarked, 0 failed. Calibrated follow-up: `bench_spark3.py`; 50/50 scored, 0 failed; merged into this file 2026-10-28T10:36:57Z.
87
 
88
+ GPQA range across candidates: 0.02–0.56 (mean 0.40). PIQA range: 0.663–0.683 (mean 0.674). Calibrated PIQA range: 0.503–0.580 (mean 0.546).
89
 
90
  ## Candidates
91
 
92
+ | trial | KL | refusals/100 | PIQA | Cal-PIQA | GPQA |
93
+ | --- | --- | --- | --- | --- | --- |
94
+ | 799 | 0.0027 | 6.0 | 0.6746 | 0.5414 | 0.50 |
95
+ | 94 | 0.0277 | 5.0 | 0.6763 | 0.5637 | 0.44 |
96
+ | 89 | 0.0289 | 4.0 | 0.6774 | 0.5630 | 0.38 |
97
+ | 177 | 0.0298 | 5.0 | 0.6703 | 0.5615 | 0.40 |
98
+ | 209 | 0.0301 | 3.0 | 0.6768 | 0.5617 | 0.26 |
99
+ | 179 | 0.0302 | 3.0 | 0.6719 | 0.5622 | 0.42 |
100
+ | 178 | 0.0309 | 6.0 | 0.6741 | 0.5647 | 0.36 |
101
+ | 175 | 0.0311 | 6.0 | 0.6687 | 0.5647 | 0.50 |
102
+ | 167 | 0.0321 | 4.0 | 0.6736 | 0.5649 | 0.44 |
103
+ | 985 | 0.0330 | 4.0 | 0.6746 | 0.5605 | 0.44 |
104
+ | 155 | 0.0362 | 2.0 | 0.6741 | 0.5595 | 0.56 |
105
+ | 189 | 0.0365 | 4.0 | 0.6774 | 0.5466 | 0.48 |
106
+ | 152 | 0.0366 | 5.0 | 0.6725 | 0.5627 | 0.42 |
107
+ | 996 | 0.0369 | 0.0 | 0.6763 | 0.5485 | 0.54 |
108
+ | 33 | 0.0387 | 5.0 | 0.6817 | 0.5448 | 0.46 |
109
+ | 160 | 0.0393 | 3.0 | 0.6741 | 0.5570 | 0.38 |
110
+ | 184 | 0.0395 | 3.0 | 0.6763 | 0.5588 | 0.52 |
111
+ | 504 | 0.0409 | 7.0 | 0.6725 | 0.5603 | 0.46 |
112
+ | 903 | 0.0450 | 0.0 | 0.6757 | 0.5520 | 0.36 |
113
+ | 861 | 0.0483 | 5.0 | 0.6774 | 0.5796 | 0.48 |
114
+ | 158 | 0.0485 | 3.0 | 0.6752 | 0.5625 | 0.40 |
115
+ | 1026 | 0.0497 | 2.0 | 0.6768 | 0.5418 | 0.30 |
116
+ | 899 | 0.0503 | 1.0 | 0.6768 | 0.5464 | 0.32 |
117
+ | 64 | 0.0529 | 4.0 | 0.6790 | 0.5446 | 0.40 |
118
+ | 945 | 0.0530 | 0.0 | 0.6741 | 0.5501 | 0.36 |
119
+ | 953 | 0.0562 | 5.0 | 0.6746 | 0.5563 | 0.46 |
120
+ | 50 | 0.0594 | 5.0 | 0.6719 | 0.5441 | 0.44 |
121
+ | 892 | 0.0877 | 8.0 | 0.6708 | 0.5533 | 0.52 |
122
+ | 63 | 0.0924 | 3.0 | 0.6785 | 0.5279 | 0.32 |
123
+ | 58 | 0.1013 | 6.0 | 0.6746 | 0.5570 | 0.52 |
124
+ | 434 | 0.1020 | 6.0 | 0.6757 | 0.5505 | 0.42 |
125
+ | 882 | 0.1138 | 0.0 | 0.6708 | 0.5555 | 0.20 |
126
+ | 376 | 0.1138 | 6.0 | 0.6719 | 0.5454 | 0.48 |
127
+ | 13 | 0.1259 | 7.0 | 0.6757 | 0.5321 | 0.50 |
128
+ | 1099 | 0.1283 | 5.0 | 0.6746 | 0.5463 | 0.54 |
129
+ | 881 | 0.1354 | 1.0 | 0.6632 | 0.5034 | 0.14 |
130
+ | 34 | 0.1477 | 4.0 | 0.6714 | 0.5625 | 0.38 |
131
+ | 1054 | 0.2197 | 0.0 | 0.6676 | 0.5349 | 0.02 |
132
+ | 16 | 0.2380 | 6.0 | 0.6692 | 0.5332 | 0.48 |
133
+ | 1077 | 0.2443 | 1.0 | 0.6703 | 0.5182 | 0.42 |
134
+ | 907 | 0.2916 | 4.0 | 0.6708 | 0.5158 | 0.48 |
135
+ | 42 | 0.3233 | 5.0 | 0.6736 | 0.5210 | 0.38 |
136
+ | 931 | 0.3271 | 5.0 | 0.6736 | 0.5237 | 0.48 |
137
+ | 15 | 0.3674 | 2.0 | 0.6817 | 0.5262 | 0.46 |
138
+ | 1078 | 0.3926 | 1.0 | 0.6834 | 0.5299 | 0.28 |
139
+ | 691 | 0.3939 | 3.0 | 0.6714 | 0.5444 | 0.38 |
140
+ | 36 | 0.4163 | 4.0 | 0.6697 | 0.5321 | 0.42 |
141
+ | 71 | 0.7175 | 4.0 | 0.6687 | 0.5337 | 0.24 |
142
+ | 1086 | 0.8243 | 0.0 | 0.6757 | 0.5158 | 0.26 |
143
+ | 23 | 1.6658 | 0.0 | 0.6719 | 0.5272 | 0.32 |
144
 
145
  ## Files
146
 
147
  - `benchmarks/results.json` - full records, correlations, config
148
  - `benchmarks/piqa_results.jsonl`, `benchmarks/gpqa_results.jsonl`
149
+ - `benchmarks/calibrated_piqa_results.jsonl` - per-candidate flip-set records
150
  - `benchmarks/partial/*.jsonl` - per-item generation checkpoints
151
  - `benchmarks/gpqa_vs_piqa.png`, `benchmarks/gpqa_vs_kld.png`
152
  - `logs/` - run logs, faults/watchdog dumps, crash reports