electroglyph/Spark-X2.6-4B-benchmarks-gpqa

Spark-X2.6-4B benchmarks: GPQA-Diamond

Which predicts GPQA better across 50 KLD-stratified Heretic trials: PIQA or low KL divergence?

Answer: inconclusive. Neither predictor beats the other. Calibrated PIQA ranks GPQA at +0.179 (fluke odds about 2 in 10): better than plain PIQA, worse than low KL, still nowhere near the bar.

Each of the 50 model variants has four scores: PIQA (commonsense quiz), calibrated PIQA (commonsense quiz, uncertain questions only), KL (how much safety tuning changed it), GPQA (hard science quiz). Rank the variants by PIQA and by KL; whichever ranking matches the GPQA ranking better is the better predictor. A later follow-up scored calibrated PIQA on the same 50 and checked it the same way; it lands between the two and changes nothing about the answer.

How the test was run:

1. Pool: 1,100 finished tuning trials from the earlier Heretic search. Each recorded a KL number and a refusal count out of 100 harmful prompts. The benchmark itself is 50 candidates x (3,676 PIQA choices + 50 GPQA questions) = 186,300 scored model calls.
2. Filtering: kept trials with at most 8 refusals and KL at most 2.0 (155 eligible), picked 50 spread evenly across KL 0.003 to 1.666.
3. Rebuild: replayed each trial's adapter onto the base model (rank 100, 5 steps, learning rate 1.0). Checked one adapter against the journal first: KL within 0.5x-1.5x of recorded, at most 2 extra refusals.
4. Commonsense quiz: full PIQA validation set, 1,838 questions with 2 answers each. Length-normalized accuracy, batches of up to 16. Base model scored 0.6763.
5. Science quiz: first 50 usable GPQA-Diamond rows in file order (198 rows, 6 skipped). Answers shuffled in fixed per-question order. One attempt each: temp 0.6, top_p 0.95, up to 16,384 new tokens, fixed seed, graded by option letter, 32-row waves on one 95 GiB GPU.
6. Win rule, fixed beforehand: lead by 0.10 or more with fluke odds under 1 in 20. KL led by 0.163 but its fluke odds were 1 in 10. No winner.
7. Calibrated follow-up: re-scored the same 50 adapters on the base model's uncertain PIQA questions only. 1,195 of 1,838 validation questions came out split across 3 samples on the base model; each candidate answered those 5 times (5,975 answers) at the model's own temperature (1.0), top_p 0.95. Base model scored 3,254/5,975. Same adapters, same replay check as step 3. This comparison was designed after the GPQA numbers were in, so its fluke odds are description, not a registered test.

What came out: the PIQA ranking barely matches GPQA; chance alone matches it that well 6 times in 10. The KL ranking matches better, but chance matches it that well 1 time in 10. The bar is 1 in 20. Neither clears it. KL looks better, but with 50 variants that gap could be luck. More variants or bigger gaps would settle it.

Calibrated PIQA (post-hoc, so the fluke odds below describe rather than test): ranking GPQA by the flip-set score agrees at +0.179, fluke odds about 2 in 10 -- a touch better than plain PIQA's +0.069, worse than low KL's +0.232, still far from the bar. The surprise is what the calibrated score does track: low KL at +0.757, fluke odds about 1 in 10 million. It barely tracks plain PIQA at all (+0.104) or refusals (+0.258). So the flip-set score behaves like a finer damage meter -- it sees the same thing KL divergence sees, with wider spread across candidates (0.503-0.580 vs PIQA's 0.663-0.683) -- not like a better GPQA predictor. Which makes sense in hindsight: the flip set is defined by the base model's uncertainty, and decensoring moves candidates off the base model's answers.

Result (n=50)

Same result as numbers. Rank agreement: -1 opposite order, 0 no relation, +1 same order. The other two columns measure the same thing two other ways and agree.

signal	rank agreement (-1 to +1)	fluke odds	second check	fluke odds	raw-score fit
PIQA vs GPQA	+0.069	0.629 (about 6 in 10)	+0.042	0.668	+0.240
low-KLD vs GPQA	+0.232	0.104 (about 1 in 10)	+0.160	0.101	+0.280
calibrated-PIQA vs GPQA	+0.179	0.211 (about 2 in 10)	+0.126	0.197	+0.309

Method

• Model: XHToken/Spark-X2.6-4B @ 66e3ae7bee54e0a523d692cb8ef841f35409807b, replayed per-trial ARA adapters (relative loss) from the Heretic journal.
• Candidates: 50 trials stratified across KL 0.003–1.666, refusals <= 8/100, KL <= 2.0.
• PIQA: full validation set, acc_norm, batch auto (same batching as the journal scores for comparability); unmodified-model baseline PIQA = 0.6763.
• GPQA: first 50 usable rows of GPQA-Diamond (Idavidrein/gpqa, 6 multi-line rows skipped), options permuted per record (permute-v1), graded by option letter; 1 run, temp 0.6, top_p 0.95, seed 20261024, max 16384 new tokens.
• Calibrated PIQA: the 1,195 validation questions the base model split on across 3 samples at its own temperature (1.0); 5 samples per candidate (base 3254/5975 = 0.5446). Harness: bench_spark3.py, scorer heretic.scorers.calibrated_piqa @ electroglyph/heretic master (369715e at scoring time); merged 2026-10-28T10:36:57Z.
• Harness: bench_spark2.py; generated 2026-10-28T04:01:10Z; 50/50 benchmarked, 0 failed. Calibrated follow-up: bench_spark3.py; 50/50 scored, 0 failed; merged into this file 2026-10-28T10:36:57Z.

GPQA range across candidates: 0.02–0.56 (mean 0.40). PIQA range: 0.663–0.683 (mean 0.674). Calibrated PIQA range: 0.503–0.580 (mean 0.546).

Candidates

trial	KL	refusals/100	PIQA	Cal-PIQA	GPQA
799	0.0027	6.0	0.6746	0.5414	0.50
94	0.0277	5.0	0.6763	0.5637	0.44
89	0.0289	4.0	0.6774	0.5630	0.38
177	0.0298	5.0	0.6703	0.5615	0.40
209	0.0301	3.0	0.6768	0.5617	0.26
179	0.0302	3.0	0.6719	0.5622	0.42
178	0.0309	6.0	0.6741	0.5647	0.36
175	0.0311	6.0	0.6687	0.5647	0.50
167	0.0321	4.0	0.6736	0.5649	0.44
985	0.0330	4.0	0.6746	0.5605	0.44
155	0.0362	2.0	0.6741	0.5595	0.56
189	0.0365	4.0	0.6774	0.5466	0.48
152	0.0366	5.0	0.6725	0.5627	0.42
996	0.0369	0.0	0.6763	0.5485	0.54
33	0.0387	5.0	0.6817	0.5448	0.46
160	0.0393	3.0	0.6741	0.5570	0.38
184	0.0395	3.0	0.6763	0.5588	0.52
504	0.0409	7.0	0.6725	0.5603	0.46
903	0.0450	0.0	0.6757	0.5520	0.36
861	0.0483	5.0	0.6774	0.5796	0.48
158	0.0485	3.0	0.6752	0.5625	0.40
1026	0.0497	2.0	0.6768	0.5418	0.30
899	0.0503	1.0	0.6768	0.5464	0.32
64	0.0529	4.0	0.6790	0.5446	0.40
945	0.0530	0.0	0.6741	0.5501	0.36
953	0.0562	5.0	0.6746	0.5563	0.46
50	0.0594	5.0	0.6719	0.5441	0.44
892	0.0877	8.0	0.6708	0.5533	0.52
63	0.0924	3.0	0.6785	0.5279	0.32
58	0.1013	6.0	0.6746	0.5570	0.52
434	0.1020	6.0	0.6757	0.5505	0.42
882	0.1138	0.0	0.6708	0.5555	0.20
376	0.1138	6.0	0.6719	0.5454	0.48
13	0.1259	7.0	0.6757	0.5321	0.50
1099	0.1283	5.0	0.6746	0.5463	0.54
881	0.1354	1.0	0.6632	0.5034	0.14
34	0.1477	4.0	0.6714	0.5625	0.38
1054	0.2197	0.0	0.6676	0.5349	0.02
16	0.2380	6.0	0.6692	0.5332	0.48
1077	0.2443	1.0	0.6703	0.5182	0.42
907	0.2916	4.0	0.6708	0.5158	0.48
42	0.3233	5.0	0.6736	0.5210	0.38
931	0.3271	5.0	0.6736	0.5237	0.48
15	0.3674	2.0	0.6817	0.5262	0.46
1078	0.3926	1.0	0.6834	0.5299	0.28
691	0.3939	3.0	0.6714	0.5444	0.38
36	0.4163	4.0	0.6697	0.5321	0.42
71	0.7175	4.0	0.6687	0.5337	0.24
1086	0.8243	0.0	0.6757	0.5158	0.26
23	1.6658	0.0	0.6719	0.5272	0.32

Files

• benchmarks/results.json - full records, correlations, config
• benchmarks/piqa_results.jsonl, benchmarks/gpqa_results.jsonl
• benchmarks/calibrated_piqa_results.jsonl - per-candidate flip-set records
• benchmarks/partial/*.jsonl - per-item generation checkpoints
• benchmarks/gpqa_vs_piqa.png, benchmarks/gpqa_vs_kld.png
• logs/ - run logs, faults/watchdog dumps, crash reports
• run_status.json - final heartbeat (phase: finished)