| # typed-decisions: single-question protocol vs full-context stress tests |
|
|
| - **single-question**: one prompt per question (the trained format). Specialist protocol: the model was trained on typed-decisions train. |
| - **full-case context**: every prompt holds the state and ALL the case questions and ends with "Answer question k"; one forward per question (batched as rows of one prefill). |
| - **full-case joint**: one prompt with all questions; answers read in sequence as "k: letter", each conditioned on the model own previous answers (one prefill + n-1 incremental steps). |
| - Neither full-case readout answers five independent decisions in one forward, and neither reproduces the public (DecisionEval) protocol: they are stress tests of request shape, not comparable scores. |
| - Canonical inference: bf16 weights under bf16 autocast. CI = bootstrap over cases. Brier and KL against the teacher distribution (dataset-card definitions). our-ECE: 10 equal-width bins of top-1 confidence, not comparable with published ECE. Score MAE: our definition (|E_p[level] - E_gold[level]|), no public code to match. |
|
|
| | model | test | protocol | correct/n | acc | 95% CI | choice | score | noul | log loss | KL | Brier | our-ECE | score MAE | |
| |---|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:| |
| | Decision-FT Stage 1 | typed_en | single-question | 1510/2000 | 0.755 | 0.734–0.776 | 0.685 | 0.745 | 0.838 | 0.646 | 0.133 | 0.070 | 0.123 | 0.242 | |
| | Decision-FT Stage 1 | typed_en | full-case context | 857/2000 | 0.428 | 0.407–0.450 | 0.330 | 0.451 | 0.497 | 1.210 | 0.496 | 0.269 | 0.111 | 0.507 | |
| | Decision-FT Stage 1 | typed_en | full-case joint | 706/2000 | 0.353 | 0.332–0.374 | 0.322 | 0.284 | 0.477 | 1.217 | 0.458 | 0.255 | 0.124 | 0.727 | |
| | Decision-FT Stage 1 | typed_es | single-question | 1482/1985 | 0.747 | 0.725–0.768 | 0.675 | 0.736 | 0.832 | 0.655 | 0.133 | 0.070 | 0.115 | 0.239 | |
| | Decision-FT Stage 1 | typed_es | full-case context | 832/1985 | 0.419 | 0.398–0.440 | 0.397 | 0.379 | 0.494 | 1.200 | 0.475 | 0.261 | 0.107 | 0.552 | |
| | Decision-FT Stage 1 | typed_es | full-case joint | 673/1985 | 0.339 | 0.317–0.360 | 0.281 | 0.280 | 0.476 | 1.204 | 0.445 | 0.247 | 0.112 | 0.690 | |
| | Decision-FT Stage 1 | telepatia_es | single-question | 1492/2000 | 0.746 | 0.725–0.768 | 0.692 | 0.719 | 0.837 | 0.677 | 0.137 | 0.072 | 0.130 | 0.261 | |
| | Decision-FT Stage 1 | telepatia_es | full-case context | 823/2000 | 0.411 | 0.390–0.433 | 0.332 | 0.416 | 0.485 | 1.190 | 0.467 | 0.260 | 0.104 | 0.530 | |
| | Decision-FT Stage 1 | telepatia_es | full-case joint | 636/2000 | 0.318 | 0.296–0.340 | 0.208 | 0.266 | 0.497 | 1.233 | 0.467 | 0.260 | 0.151 | 0.687 | |
| | Ines-1 (release) | typed_en | single-question | 1528/2000 | 0.764 | 0.742–0.785 | 0.715 | 0.745 | 0.838 | 0.635 | 0.121 | 0.065 | 0.132 | 0.234 | |
| | Ines-1 (release) | typed_en | full-case context | 831/2000 | 0.415 | 0.395–0.436 | 0.255 | 0.479 | 0.492 | 1.195 | 0.487 | 0.266 | 0.118 | 0.540 | |
| | Ines-1 (release) | typed_en | full-case joint | 742/2000 | 0.371 | 0.349–0.393 | 0.287 | 0.367 | 0.460 | 1.182 | 0.437 | 0.246 | 0.089 | 0.618 | |
| | Ines-1 (release) | typed_es | single-question | 1496/1985 | 0.754 | 0.732–0.774 | 0.699 | 0.739 | 0.827 | 0.639 | 0.119 | 0.065 | 0.123 | 0.236 | |
| | Ines-1 (release) | typed_es | full-case context | 859/1985 | 0.433 | 0.411–0.454 | 0.355 | 0.443 | 0.496 | 1.162 | 0.450 | 0.251 | 0.099 | 0.546 | |
| | Ines-1 (release) | typed_es | full-case joint | 763/1985 | 0.384 | 0.362–0.407 | 0.379 | 0.317 | 0.479 | 1.172 | 0.421 | 0.233 | 0.077 | 0.678 | |
| | Ines-1 (release) | telepatia_es | single-question | 1526/2000 | 0.763 | 0.742–0.785 | 0.730 | 0.738 | 0.830 | 0.639 | 0.119 | 0.064 | 0.134 | 0.236 | |
| | Ines-1 (release) | telepatia_es | full-case context | 878/2000 | 0.439 | 0.418–0.460 | 0.372 | 0.449 | 0.493 | 1.147 | 0.441 | 0.247 | 0.092 | 0.539 | |
| | Ines-1 (release) | telepatia_es | full-case joint | 745/2000 | 0.372 | 0.350–0.395 | 0.383 | 0.295 | 0.465 | 1.175 | 0.424 | 0.236 | 0.083 | 0.672 | |
|
|
| Dataset reference (card): Prior 0.470 accuracy, Brier 0.189. |
|
|
| ## Paired (exact McNemar, same questions) |
|
|
| | model | test | comparison | n | only first right | only second right | p | |
| |---|---|---|---:|---:|---:|---:| |
| | Decision-FT Stage 1 | typed_en | single-question vs full-case context | 2000 | 797 | 144 | 3.6e-110 | |
| | Decision-FT Stage 1 | typed_en | full-case context vs full-case joint | 2000 | 453 | 302 | 4.3e-08 | |
| | Decision-FT Stage 1 | typed_es | single-question vs full-case context | 1985 | 769 | 119 | 3.8e-117 | |
| | Decision-FT Stage 1 | typed_es | full-case context vs full-case joint | 1985 | 413 | 254 | 7.9e-10 | |
| | Decision-FT Stage 1 | telepatia_es | single-question vs full-case context | 2000 | 795 | 126 | 2.3e-119 | |
| | Decision-FT Stage 1 | telepatia_es | full-case context vs full-case joint | 2000 | 448 | 261 | 2.1e-12 | |
| | Ines-1 (release) | typed_en | single-question vs full-case context | 2000 | 847 | 150 | 1.5e-118 | |
| | Ines-1 (release) | typed_en | full-case context vs full-case joint | 2000 | 474 | 385 | 0.0027 | |
| | Ines-1 (release) | typed_es | single-question vs full-case context | 1985 | 774 | 137 | 1.7e-108 | |
| | Ines-1 (release) | typed_es | full-case context vs full-case joint | 1985 | 416 | 320 | 0.00045 | |
| | Ines-1 (release) | telepatia_es | single-question vs full-case context | 2000 | 784 | 136 | 2.4e-111 | |
| | Ines-1 (release) | telepatia_es | full-case context vs full-case joint | 2000 | 463 | 330 | 2.6e-06 | |
|
|
