eval/report.md

Audit report

Names: v2 = Decision-FT Stage 1; v3 = Ines-1 (the release, codename mini-v41-Decisions); v3pre = Decision-Mix Stage 2 from Pretrain-V3 (ablation).

Accuracy and calibration

model	test	correct/n	acc	95% CI (cases)	choice	score	noul	NLL	KL(gold‖p)	Brier (soft)	ECE10 (ours)
v2	tasksource	850/2206	0.385	0.365–0.407	0.366	0.269	0.579	1.182	1.158	0.658	0.126
v2	telepatia_es	1492/2000	0.746	0.725–0.768	0.692	0.719	0.837	0.677	0.137	0.072	0.130
v2	typed_en	1510/2000	0.755	0.734–0.776	0.685	0.745	0.838	0.646	0.133	0.070	0.123
v2	typed_es	1482/1985	0.747	0.725–0.768	0.675	0.736	0.832	0.655	0.133	0.070	0.115
v2-engram_off	tasksource	816/2206	0.370	0.350–0.390	0.349	0.277	0.559	1.178	1.154	0.655	0.132
v2-engram_off	telepatia_es	1433/2000	0.717	0.694–0.739	0.673	0.669	0.823	0.735	0.157	0.084	0.133
v2-engram_off	typed_en	1479/2000	0.740	0.718–0.761	0.680	0.714	0.833	0.685	0.140	0.074	0.129
v2-engram_off	typed_es	1427/1985	0.719	0.697–0.741	0.648	0.694	0.822	0.713	0.150	0.080	0.125
v3	tasksource	1093/2206	0.495	0.474–0.516	0.479	0.392	0.659	1.030	1.008	0.567	0.071
v3	telepatia_es	1526/2000	0.763	0.742–0.785	0.730	0.738	0.830	0.639	0.119	0.064	0.134
v3	typed_en	1528/2000	0.764	0.742–0.785	0.715	0.745	0.838	0.635	0.121	0.065	0.132
v3	typed_es	1496/1985	0.754	0.732–0.774	0.699	0.739	0.827	0.639	0.119	0.065	0.123
v3-engram_off	tasksource	1037/2206	0.470	0.449–0.490	0.457	0.400	0.598	1.040	1.017	0.575	0.069
v3-engram_off	telepatia_es	1469/2000	0.735	0.713–0.756	0.668	0.713	0.830	0.692	0.137	0.073	0.133
v3-engram_off	typed_en	1494/2000	0.747	0.726–0.768	0.683	0.741	0.818	0.666	0.130	0.070	0.130
v3-engram_off	typed_es	1446/1985	0.728	0.706–0.750	0.663	0.705	0.824	0.696	0.137	0.074	0.120
v3pre	tasksource	1118/2206	0.507	0.485–0.527	0.492	0.446	0.640	1.013	0.990	0.558	0.068
v3pre	telepatia_es	1423/2000	0.712	0.690–0.732	0.635	0.711	0.788	0.722	0.164	0.092	0.110
v3pre	typed_en	1451/2000	0.726	0.705–0.746	0.655	0.714	0.812	0.723	0.158	0.087	0.131
v3pre	typed_es	1397/1985	0.704	0.682–0.725	0.614	0.702	0.796	0.724	0.162	0.090	0.101
v3pre-engram_off	tasksource	1083/2206	0.491	0.470–0.512	0.473	0.408	0.659	1.036	1.013	0.572	0.071
v3pre-engram_off	telepatia_es	1369/2000	0.684	0.663–0.706	0.627	0.664	0.770	0.766	0.184	0.103	0.102
v3pre-engram_off	typed_en	1366/2000	0.683	0.662–0.704	0.635	0.654	0.770	0.770	0.184	0.104	0.101
v3pre-engram_off	typed_es	1337/1985	0.674	0.652–0.696	0.596	0.656	0.774	0.771	0.184	0.104	0.092

Paired comparisons (same questions, exact McNemar)

comparison	n	only first right	only second right	p
v2 vs v3	typed_en	2000	97	115
v2 vs v3pre	typed_en	2000	204	145
v3 vs v3pre	typed_en	2000	216	139
v2 vs v2-engram_off	typed_en	2000	118	87
v3 vs v3-engram_off	typed_en	2000	114	80
v3pre vs v3pre-engram_off	typed_en	2000	169	84
v2 vs v3	typed_es	1985	79	93
v2 vs v3pre	typed_es	1985	225	140
v3 vs v3pre	typed_es	1985	223	124
v2 vs v2-engram_off	typed_es	1985	139	84
v3 vs v3-engram_off	typed_es	1985	136	86
v3pre vs v3pre-engram_off	typed_es	1985	172	112
v2 vs v3	telepatia_es	2000	89	123
v2 vs v3pre	telepatia_es	2000	211	142
v3 vs v3pre	telepatia_es	2000	230	127
v2 vs v2-engram_off	telepatia_es	2000	132	73
v3 vs v3-engram_off	telepatia_es	2000	149	92
v3pre vs v3pre-engram_off	telepatia_es	2000	156	102
v2 vs v3	tasksource	2206	376	619
v2 vs v3pre	tasksource	2206	383	651
v3 vs v3pre	tasksource	2206	213	238
v2 vs v2-engram_off	tasksource	2206	151	117
v3 vs v3-engram_off	tasksource	2206	234	178
v3pre vs v3pre-engram_off	tasksource	2206	222	187