benchmarks/README.md

Benchmark summaries

All numbers here were measured locally on one RTX 5060 Ti 16GB using the same rows and scorer for every model.

Typed Decisions

• Full independent test split: 400 held-out cases, 2,000 probabilistic decisions.
• Scored with the benchmark metric code: argmax accuracy, KL, Brier, ECE.
• Models: Pluto-2.1, Pluto-2, d1-2B, d1-3B. d1 checkpoints are zero-shot.

DecisionBench

• Public Hanno-Labs/decision-bench evaluation split.
• Matched accessibility subset: 1,565 of 23,900 rows, limited to ≤16 candidates and ≤2,048-token prompts.
• Models: Pluto-2.1, Pluto-2, Bosun 3.1 1.7B, d1-2B, d1-3B.
• This is not the full 23,900-row suite. No full-suite score is claimed.

Files

• pluto21_*.json — Pluto-2.1 step 2,000
• pluto2_*.json — Pluto-2 400M
• bosun_base_decisionbench_1565.json — Bosun 3.1 1.7B initializer
• d1_2b_*.json — Liquid d1-2B
• d1_3b_*.json — Liquid d1-3B

The model card charts are generated from these values.