Hugging Face's logo

Endikavi
/
Ines-1

Safetensors
English
Spanish
mini-v41
mixture-of-experts
decision-model
typed-decisions
classification
english
spanish
Model card Files
xet
 Community
Ines-1 / eval /fullcase_report.md
Endikavi's picture
Endikavi
Ines-1 RC1 (private staging; release commit b7f5644)
61b6fb9 verified
preview code
|
Raw History Blame Contribute Delete
5.5 kB

typed-decisions: single-question protocol vs full-context stress tests

  • single-question: one prompt per question (the trained format). Specialist protocol: the model was trained on typed-decisions train.
  • full-case context: every prompt holds the state and ALL the case questions and ends with "Answer question k"; one forward per question (batched as rows of one prefill).
  • full-case joint: one prompt with all questions; answers read in sequence as "k: letter", each conditioned on the model own previous answers (one prefill + n-1 incremental steps).
  • Neither full-case readout answers five independent decisions in one forward, and neither reproduces the public (DecisionEval) protocol: they are stress tests of request shape, not comparable scores.
  • Canonical inference: bf16 weights under bf16 autocast. CI = bootstrap over cases. Brier and KL against the teacher distribution (dataset-card definitions). our-ECE: 10 equal-width bins of top-1 confidence, not comparable with published ECE. Score MAE: our definition (|E_p[level] - E_gold[level]|), no public code to match.
model test protocol correct/n acc 95% CI choice score noul log loss KL Brier our-ECE score MAE
Decision-FT Stage 1 typed_en single-question 1510/2000 0.755 0.734–0.776 0.685 0.745 0.838 0.646 0.133 0.070 0.123 0.242
Decision-FT Stage 1 typed_en full-case context 857/2000 0.428 0.407–0.450 0.330 0.451 0.497 1.210 0.496 0.269 0.111 0.507
Decision-FT Stage 1 typed_en full-case joint 706/2000 0.353 0.332–0.374 0.322 0.284 0.477 1.217 0.458 0.255 0.124 0.727
Decision-FT Stage 1 typed_es single-question 1482/1985 0.747 0.725–0.768 0.675 0.736 0.832 0.655 0.133 0.070 0.115 0.239
Decision-FT Stage 1 typed_es full-case context 832/1985 0.419 0.398–0.440 0.397 0.379 0.494 1.200 0.475 0.261 0.107 0.552
Decision-FT Stage 1 typed_es full-case joint 673/1985 0.339 0.317–0.360 0.281 0.280 0.476 1.204 0.445 0.247 0.112 0.690
Decision-FT Stage 1 telepatia_es single-question 1492/2000 0.746 0.725–0.768 0.692 0.719 0.837 0.677 0.137 0.072 0.130 0.261
Decision-FT Stage 1 telepatia_es full-case context 823/2000 0.411 0.390–0.433 0.332 0.416 0.485 1.190 0.467 0.260 0.104 0.530
Decision-FT Stage 1 telepatia_es full-case joint 636/2000 0.318 0.296–0.340 0.208 0.266 0.497 1.233 0.467 0.260 0.151 0.687
Ines-1 (release) typed_en single-question 1528/2000 0.764 0.742–0.785 0.715 0.745 0.838 0.635 0.121 0.065 0.132 0.234
Ines-1 (release) typed_en full-case context 831/2000 0.415 0.395–0.436 0.255 0.479 0.492 1.195 0.487 0.266 0.118 0.540
Ines-1 (release) typed_en full-case joint 742/2000 0.371 0.349–0.393 0.287 0.367 0.460 1.182 0.437 0.246 0.089 0.618
Ines-1 (release) typed_es single-question 1496/1985 0.754 0.732–0.774 0.699 0.739 0.827 0.639 0.119 0.065 0.123 0.236
Ines-1 (release) typed_es full-case context 859/1985 0.433 0.411–0.454 0.355 0.443 0.496 1.162 0.450 0.251 0.099 0.546
Ines-1 (release) typed_es full-case joint 763/1985 0.384 0.362–0.407 0.379 0.317 0.479 1.172 0.421 0.233 0.077 0.678
Ines-1 (release) telepatia_es single-question 1526/2000 0.763 0.742–0.785 0.730 0.738 0.830 0.639 0.119 0.064 0.134 0.236
Ines-1 (release) telepatia_es full-case context 878/2000 0.439 0.418–0.460 0.372 0.449 0.493 1.147 0.441 0.247 0.092 0.539
Ines-1 (release) telepatia_es full-case joint 745/2000 0.372 0.350–0.395 0.383 0.295 0.465 1.175 0.424 0.236 0.083 0.672

Dataset reference (card): Prior 0.470 accuracy, Brier 0.189.

Paired (exact McNemar, same questions)

model test comparison n only first right only second right p
Decision-FT Stage 1 typed_en single-question vs full-case context 2000 797 144 3.6e-110
Decision-FT Stage 1 typed_en full-case context vs full-case joint 2000 453 302 4.3e-08
Decision-FT Stage 1 typed_es single-question vs full-case context 1985 769 119 3.8e-117
Decision-FT Stage 1 typed_es full-case context vs full-case joint 1985 413 254 7.9e-10
Decision-FT Stage 1 telepatia_es single-question vs full-case context 2000 795 126 2.3e-119
Decision-FT Stage 1 telepatia_es full-case context vs full-case joint 2000 448 261 2.1e-12
Ines-1 (release) typed_en single-question vs full-case context 2000 847 150 1.5e-118
Ines-1 (release) typed_en full-case context vs full-case joint 2000 474 385 0.0027
Ines-1 (release) typed_es single-question vs full-case context 1985 774 137 1.7e-108
Ines-1 (release) typed_es full-case context vs full-case joint 1985 416 320 0.00045
Ines-1 (release) telepatia_es single-question vs full-case context 2000 784 136 2.4e-111
Ines-1 (release) telepatia_es full-case context vs full-case joint 2000 463 330 2.6e-06