Ines-1 / eval /report.md

Ines-1 RC1 (private staging; release commit b7f5644)

61b6fb9 verified 26 days ago

4.76 kB

# Audit report

Names: v2 = Decision-FT Stage 1; v3 = Ines-1 (the release, codename mini-v41-Decisions); v3pre = Decision-Mix Stage 2 from Pretrain-V3 (ablation).

## Accuracy and calibration

| model | test | correct/n | acc | 95% CI (cases) | choice | score | noul | NLL | KL(gold‖p) | Brier (soft) | ECE10 (ours) |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|
| v2 | tasksource | 850/2206 | 0.385 | 0.365–0.407 | 0.366 | 0.269 | 0.579 | 1.182 | 1.158 | 0.658 | 0.126 |
| v2 | telepatia_es | 1492/2000 | 0.746 | 0.725–0.768 | 0.692 | 0.719 | 0.837 | 0.677 | 0.137 | 0.072 | 0.130 |
| v2 | typed_en | 1510/2000 | 0.755 | 0.734–0.776 | 0.685 | 0.745 | 0.838 | 0.646 | 0.133 | 0.070 | 0.123 |
| v2 | typed_es | 1482/1985 | 0.747 | 0.725–0.768 | 0.675 | 0.736 | 0.832 | 0.655 | 0.133 | 0.070 | 0.115 |
| v2-engram_off | tasksource | 816/2206 | 0.370 | 0.350–0.390 | 0.349 | 0.277 | 0.559 | 1.178 | 1.154 | 0.655 | 0.132 |
| v2-engram_off | telepatia_es | 1433/2000 | 0.717 | 0.694–0.739 | 0.673 | 0.669 | 0.823 | 0.735 | 0.157 | 0.084 | 0.133 |
| v2-engram_off | typed_en | 1479/2000 | 0.740 | 0.718–0.761 | 0.680 | 0.714 | 0.833 | 0.685 | 0.140 | 0.074 | 0.129 |
| v2-engram_off | typed_es | 1427/1985 | 0.719 | 0.697–0.741 | 0.648 | 0.694 | 0.822 | 0.713 | 0.150 | 0.080 | 0.125 |
| v3 | tasksource | 1093/2206 | 0.495 | 0.474–0.516 | 0.479 | 0.392 | 0.659 | 1.030 | 1.008 | 0.567 | 0.071 |
| v3 | telepatia_es | 1526/2000 | 0.763 | 0.742–0.785 | 0.730 | 0.738 | 0.830 | 0.639 | 0.119 | 0.064 | 0.134 |
| v3 | typed_en | 1528/2000 | 0.764 | 0.742–0.785 | 0.715 | 0.745 | 0.838 | 0.635 | 0.121 | 0.065 | 0.132 |
| v3 | typed_es | 1496/1985 | 0.754 | 0.732–0.774 | 0.699 | 0.739 | 0.827 | 0.639 | 0.119 | 0.065 | 0.123 |
| v3-engram_off | tasksource | 1037/2206 | 0.470 | 0.449–0.490 | 0.457 | 0.400 | 0.598 | 1.040 | 1.017 | 0.575 | 0.069 |
| v3-engram_off | telepatia_es | 1469/2000 | 0.735 | 0.713–0.756 | 0.668 | 0.713 | 0.830 | 0.692 | 0.137 | 0.073 | 0.133 |
| v3-engram_off | typed_en | 1494/2000 | 0.747 | 0.726–0.768 | 0.683 | 0.741 | 0.818 | 0.666 | 0.130 | 0.070 | 0.130 |
| v3-engram_off | typed_es | 1446/1985 | 0.728 | 0.706–0.750 | 0.663 | 0.705 | 0.824 | 0.696 | 0.137 | 0.074 | 0.120 |
| v3pre | tasksource | 1118/2206 | 0.507 | 0.485–0.527 | 0.492 | 0.446 | 0.640 | 1.013 | 0.990 | 0.558 | 0.068 |
| v3pre | telepatia_es | 1423/2000 | 0.712 | 0.690–0.732 | 0.635 | 0.711 | 0.788 | 0.722 | 0.164 | 0.092 | 0.110 |
| v3pre | typed_en | 1451/2000 | 0.726 | 0.705–0.746 | 0.655 | 0.714 | 0.812 | 0.723 | 0.158 | 0.087 | 0.131 |
| v3pre | typed_es | 1397/1985 | 0.704 | 0.682–0.725 | 0.614 | 0.702 | 0.796 | 0.724 | 0.162 | 0.090 | 0.101 |
| v3pre-engram_off | tasksource | 1083/2206 | 0.491 | 0.470–0.512 | 0.473 | 0.408 | 0.659 | 1.036 | 1.013 | 0.572 | 0.071 |
| v3pre-engram_off | telepatia_es | 1369/2000 | 0.684 | 0.663–0.706 | 0.627 | 0.664 | 0.770 | 0.766 | 0.184 | 0.103 | 0.102 |
| v3pre-engram_off | typed_en | 1366/2000 | 0.683 | 0.662–0.704 | 0.635 | 0.654 | 0.770 | 0.770 | 0.184 | 0.104 | 0.101 |
| v3pre-engram_off | typed_es | 1337/1985 | 0.674 | 0.652–0.696 | 0.596 | 0.656 | 0.774 | 0.771 | 0.184 | 0.104 | 0.092 |

## Paired comparisons (same questions, exact McNemar)

| comparison | n | only first right | only second right | p |
|---|---:|---:|---:|---:|
| v2 vs v3|typed_en | 2000 | 97 | 115 | 0.24 |
| v2 vs v3pre|typed_en | 2000 | 204 | 145 | 0.0019 |
| v3 vs v3pre|typed_en | 2000 | 216 | 139 | 5.2e-05 |
| v2 vs v2-engram_off|typed_en | 2000 | 118 | 87 | 0.036 |
| v3 vs v3-engram_off|typed_en | 2000 | 114 | 80 | 0.018 |
| v3pre vs v3pre-engram_off|typed_en | 2000 | 169 | 84 | 9.8e-08 |
| v2 vs v3|typed_es | 1985 | 79 | 93 | 0.32 |
| v2 vs v3pre|typed_es | 1985 | 225 | 140 | 1e-05 |
| v3 vs v3pre|typed_es | 1985 | 223 | 124 | 1.2e-07 |
| v2 vs v2-engram_off|typed_es | 1985 | 139 | 84 | 0.00028 |
| v3 vs v3-engram_off|typed_es | 1985 | 136 | 86 | 0.00096 |
| v3pre vs v3pre-engram_off|typed_es | 1985 | 172 | 112 | 0.00044 |
| v2 vs v3|telepatia_es | 2000 | 89 | 123 | 0.023 |
| v2 vs v3pre|telepatia_es | 2000 | 211 | 142 | 0.00028 |
| v3 vs v3pre|telepatia_es | 2000 | 230 | 127 | 5.5e-08 |
| v2 vs v2-engram_off|telepatia_es | 2000 | 132 | 73 | 4.6e-05 |
| v3 vs v3-engram_off|telepatia_es | 2000 | 149 | 92 | 0.00029 |
| v3pre vs v3pre-engram_off|telepatia_es | 2000 | 156 | 102 | 0.00093 |
| v2 vs v3|tasksource | 2206 | 376 | 619 | 1.3e-14 |
| v2 vs v3pre|tasksource | 2206 | 383 | 651 | 6.8e-17 |
| v3 vs v3pre|tasksource | 2206 | 213 | 238 | 0.26 |
| v2 vs v2-engram_off|tasksource | 2206 | 151 | 117 | 0.044 |
| v3 vs v3-engram_off|tasksource | 2206 | 234 | 178 | 0.0067 |
| v3pre vs v3pre-engram_off|tasksource | 2206 | 222 | 187 | 0.093 |

	# Audit report

	Names: v2 = Decision-FT Stage 1; v3 = Ines-1 (the release, codename mini-v41-Decisions); v3pre = Decision-Mix Stage 2 from Pretrain-V3 (ablation).

	## Accuracy and calibration

	\| model \| test \| correct/n \| acc \| 95% CI (cases) \| choice \| score \| noul \| NLL \| KL(gold‖p) \| Brier (soft) \| ECE10 (ours) \|
	\|---\|---\|---:\|---:\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| v2 \| tasksource \| 850/2206 \| 0.385 \| 0.365–0.407 \| 0.366 \| 0.269 \| 0.579 \| 1.182 \| 1.158 \| 0.658 \| 0.126 \|
	\| v2 \| telepatia_es \| 1492/2000 \| 0.746 \| 0.725–0.768 \| 0.692 \| 0.719 \| 0.837 \| 0.677 \| 0.137 \| 0.072 \| 0.130 \|
	\| v2 \| typed_en \| 1510/2000 \| 0.755 \| 0.734–0.776 \| 0.685 \| 0.745 \| 0.838 \| 0.646 \| 0.133 \| 0.070 \| 0.123 \|
	\| v2 \| typed_es \| 1482/1985 \| 0.747 \| 0.725–0.768 \| 0.675 \| 0.736 \| 0.832 \| 0.655 \| 0.133 \| 0.070 \| 0.115 \|
	\| v2-engram_off \| tasksource \| 816/2206 \| 0.370 \| 0.350–0.390 \| 0.349 \| 0.277 \| 0.559 \| 1.178 \| 1.154 \| 0.655 \| 0.132 \|
	\| v2-engram_off \| telepatia_es \| 1433/2000 \| 0.717 \| 0.694–0.739 \| 0.673 \| 0.669 \| 0.823 \| 0.735 \| 0.157 \| 0.084 \| 0.133 \|
	\| v2-engram_off \| typed_en \| 1479/2000 \| 0.740 \| 0.718–0.761 \| 0.680 \| 0.714 \| 0.833 \| 0.685 \| 0.140 \| 0.074 \| 0.129 \|
	\| v2-engram_off \| typed_es \| 1427/1985 \| 0.719 \| 0.697–0.741 \| 0.648 \| 0.694 \| 0.822 \| 0.713 \| 0.150 \| 0.080 \| 0.125 \|
	\| v3 \| tasksource \| 1093/2206 \| 0.495 \| 0.474–0.516 \| 0.479 \| 0.392 \| 0.659 \| 1.030 \| 1.008 \| 0.567 \| 0.071 \|
	\| v3 \| telepatia_es \| 1526/2000 \| 0.763 \| 0.742–0.785 \| 0.730 \| 0.738 \| 0.830 \| 0.639 \| 0.119 \| 0.064 \| 0.134 \|
	\| v3 \| typed_en \| 1528/2000 \| 0.764 \| 0.742–0.785 \| 0.715 \| 0.745 \| 0.838 \| 0.635 \| 0.121 \| 0.065 \| 0.132 \|
	\| v3 \| typed_es \| 1496/1985 \| 0.754 \| 0.732–0.774 \| 0.699 \| 0.739 \| 0.827 \| 0.639 \| 0.119 \| 0.065 \| 0.123 \|
	\| v3-engram_off \| tasksource \| 1037/2206 \| 0.470 \| 0.449–0.490 \| 0.457 \| 0.400 \| 0.598 \| 1.040 \| 1.017 \| 0.575 \| 0.069 \|
	\| v3-engram_off \| telepatia_es \| 1469/2000 \| 0.735 \| 0.713–0.756 \| 0.668 \| 0.713 \| 0.830 \| 0.692 \| 0.137 \| 0.073 \| 0.133 \|
	\| v3-engram_off \| typed_en \| 1494/2000 \| 0.747 \| 0.726–0.768 \| 0.683 \| 0.741 \| 0.818 \| 0.666 \| 0.130 \| 0.070 \| 0.130 \|
	\| v3-engram_off \| typed_es \| 1446/1985 \| 0.728 \| 0.706–0.750 \| 0.663 \| 0.705 \| 0.824 \| 0.696 \| 0.137 \| 0.074 \| 0.120 \|
	\| v3pre \| tasksource \| 1118/2206 \| 0.507 \| 0.485–0.527 \| 0.492 \| 0.446 \| 0.640 \| 1.013 \| 0.990 \| 0.558 \| 0.068 \|
	\| v3pre \| telepatia_es \| 1423/2000 \| 0.712 \| 0.690–0.732 \| 0.635 \| 0.711 \| 0.788 \| 0.722 \| 0.164 \| 0.092 \| 0.110 \|
	\| v3pre \| typed_en \| 1451/2000 \| 0.726 \| 0.705–0.746 \| 0.655 \| 0.714 \| 0.812 \| 0.723 \| 0.158 \| 0.087 \| 0.131 \|
	\| v3pre \| typed_es \| 1397/1985 \| 0.704 \| 0.682–0.725 \| 0.614 \| 0.702 \| 0.796 \| 0.724 \| 0.162 \| 0.090 \| 0.101 \|
	\| v3pre-engram_off \| tasksource \| 1083/2206 \| 0.491 \| 0.470–0.512 \| 0.473 \| 0.408 \| 0.659 \| 1.036 \| 1.013 \| 0.572 \| 0.071 \|
	\| v3pre-engram_off \| telepatia_es \| 1369/2000 \| 0.684 \| 0.663–0.706 \| 0.627 \| 0.664 \| 0.770 \| 0.766 \| 0.184 \| 0.103 \| 0.102 \|
	\| v3pre-engram_off \| typed_en \| 1366/2000 \| 0.683 \| 0.662–0.704 \| 0.635 \| 0.654 \| 0.770 \| 0.770 \| 0.184 \| 0.104 \| 0.101 \|
	\| v3pre-engram_off \| typed_es \| 1337/1985 \| 0.674 \| 0.652–0.696 \| 0.596 \| 0.656 \| 0.774 \| 0.771 \| 0.184 \| 0.104 \| 0.092 \|

	## Paired comparisons (same questions, exact McNemar)

	\| comparison \| n \| only first right \| only second right \| p \|
	\|---\|---:\|---:\|---:\|---:\|
	\| v2 vs v3\|typed_en \| 2000 \| 97 \| 115 \| 0.24 \|
	\| v2 vs v3pre\|typed_en \| 2000 \| 204 \| 145 \| 0.0019 \|
	\| v3 vs v3pre\|typed_en \| 2000 \| 216 \| 139 \| 5.2e-05 \|
	\| v2 vs v2-engram_off\|typed_en \| 2000 \| 118 \| 87 \| 0.036 \|
	\| v3 vs v3-engram_off\|typed_en \| 2000 \| 114 \| 80 \| 0.018 \|
	\| v3pre vs v3pre-engram_off\|typed_en \| 2000 \| 169 \| 84 \| 9.8e-08 \|
	\| v2 vs v3\|typed_es \| 1985 \| 79 \| 93 \| 0.32 \|
	\| v2 vs v3pre\|typed_es \| 1985 \| 225 \| 140 \| 1e-05 \|
	\| v3 vs v3pre\|typed_es \| 1985 \| 223 \| 124 \| 1.2e-07 \|
	\| v2 vs v2-engram_off\|typed_es \| 1985 \| 139 \| 84 \| 0.00028 \|
	\| v3 vs v3-engram_off\|typed_es \| 1985 \| 136 \| 86 \| 0.00096 \|
	\| v3pre vs v3pre-engram_off\|typed_es \| 1985 \| 172 \| 112 \| 0.00044 \|
	\| v2 vs v3\|telepatia_es \| 2000 \| 89 \| 123 \| 0.023 \|
	\| v2 vs v3pre\|telepatia_es \| 2000 \| 211 \| 142 \| 0.00028 \|
	\| v3 vs v3pre\|telepatia_es \| 2000 \| 230 \| 127 \| 5.5e-08 \|
	\| v2 vs v2-engram_off\|telepatia_es \| 2000 \| 132 \| 73 \| 4.6e-05 \|
	\| v3 vs v3-engram_off\|telepatia_es \| 2000 \| 149 \| 92 \| 0.00029 \|
	\| v3pre vs v3pre-engram_off\|telepatia_es \| 2000 \| 156 \| 102 \| 0.00093 \|
	\| v2 vs v3\|tasksource \| 2206 \| 376 \| 619 \| 1.3e-14 \|
	\| v2 vs v3pre\|tasksource \| 2206 \| 383 \| 651 \| 6.8e-17 \|
	\| v3 vs v3pre\|tasksource \| 2206 \| 213 \| 238 \| 0.26 \|
	\| v2 vs v2-engram_off\|tasksource \| 2206 \| 151 \| 117 \| 0.044 \|
	\| v3 vs v3-engram_off\|tasksource \| 2206 \| 234 \| 178 \| 0.0067 \|
	\| v3pre vs v3pre-engram_off\|tasksource \| 2206 \| 222 \| 187 \| 0.093 \|