Inkling-2-Mini Decision (eltokh7/inkling-2-mini-decision)

Should we have System 1.5 models?

An open-weights System 1.5 adaptive-depth decision model built on thinkingmachines/Inkling-2-Mini (26.4B total / 3.4Bโ€“5.4B active MoE, 24 physical layers with a 12-layer looped core).

Instead of generating prose or JSON tokens, it evaluates a state and typed questions (choice, noul, score) in a single forward pass and dynamically decides how many internal loops (1, 2, or 3 = 24, 36, or 48 logical layers) to spend before answering โ€” with zero reasoning tokens.

Key Results

  • Routine decisions (decision-v7/test, $n=1{,}440$): 85.56% accuracy, 0.1944 Brier, 0.0235 ECE โ€” right alongside 27B-dense Kev-27B (86.3%) at ~5ร— lower active prefill FLOPs.
  • Difficulty-routed early exit (ฯ„ = 0.90): 54.2% of routine decision-v7 questions exit right after Loop 1 (only 8.8% of hard OOD hard-v1 questions do โ€” 76.8% automatically route to Loop 3).
  • Real GPU wall-clock speed (merge_and_unload + torch.compile, RTX PRO 6000 Blackwell, median headline / mean in parens):
    • Batch-1 Early Exit (ฯ„ = 0.90): 81.4 ms median (97.4 ms mean) โ€” 1.68ร— faster median than compiled Loop 3 (136.8 ms median / 146.3 ms mean) with โˆ’0.07 pp accuracy difference (85.49% vs. 85.56%).
    • Batch-1 Early Exit (ฯ„ = 0.85): 79.6 ms median (95.4 ms mean) โ€” 1.72ร— faster median.
    • Per-pass medians (batch-1): 65.6 ms (Loop 1) / 100.2 ms (Loop 2) / 136.8 ms (Loop 3).
    • Batched throughput (bs = 8, causal_conv1d): 27.9 ms/q (35.8 q/s) at Loop 1 and 61.4 ms/q (16.3 q/s) at Loop 3.
  • Latent reasoning on untrained OOD hard tasks (hard-v1): Going from Loop 1 โ†’ Loop 3 adds +26.8 pp on hard-v1/dev (39.9% โ†’ 66.7%) and +24.3 pp on hard-v1/test (42.6% โ†’ 66.8%), beating 9B-dense Kev-9B v1 (58.2% / 58.4%) by +8.4 pp with 5.4B active parameters.

Comparison on untrained hard-v1

Architecture

  1. Frozen 26.4B MoE + Attention LoRA (r = 32): LoRA adapters (17.5M params) attached to self_attn.{q,k,v,o}_proj.{linear,gate_proj}; lm_head and vision_tower dropped for text decisioning.
  2. Zero-Waste Layer-17 Halting Gate (CoreGateHead, 1.2M params): After Loop 1 or Loop 2, a lightweight bilinear probe (~0.05 ms) reads the Layer-17 core output before running exit layers 18โ€“23. If calibrated confidence clears threshold, it skips the remaining loops and runs exit layers once.
  3. Two-Stage JointSchemaHead (6.0M params):
    • Stage 1 (Evidence Routing): <decide> and every </opt> token cross-attend to 8 summary tokens of the input state.
    • Stage 2 (Cross-Option Attention): Options and <decide> self-attend to compare directly before bilinear pointer scoring.
  4. Multi-Loop Deep Supervision: Trained jointly across Loops 1, 2, and 3 (0.2ยทL1 + 0.3ยทL2 + 0.5ยทL3) using Label-Smoothed Cross-Entropy + Brier Loss (0.5), then temperature-calibrated on decision-v7/calibration.

Quickstart

from modeling import InklingDecisionModel

model = InklingDecisionModel.from_pretrained("eltokh7/inkling-2-mini-decision")

out = model.decide(
    state="Customer email: 'My card was charged twice for order #4821 yesterday. Please refund.'",
    questions={
        "route": {
            "type": "choice",
            "instructions": "Which team should handle this ticket?",
            "criteria": {
                "billing": "Duplicate charges, refunds, invoices",
                "tech": "Bugs, login issues, crashes",
                "sales": "Pricing, upgrades, quotes",
            },
        },
        "urgent": {
            "type": "noul",
            "instructions": "Does this require immediate escalation?",
        },
    },
    threshold=0.90,  # set to 1.01 to always run all 3 loops
)
print(out)

Benchmarks

1. Loop 1 vs. Loop 2 vs. Loop 3

Split $n$ Loop 1 (24L) Loop 2 (36L) Loop 3 (48L) L3 Brier L3 ECE ฮ” (L3 โˆ’ L1)
decision-v7/dev 1,468 81.95% 85.63% 85.63% 0.2056 0.0243 +3.68 pp
decision-v7/test 1,440 82.29% 84.93% 85.56% 0.1944 0.0235 +3.27 pp
hard-v1/dev (untrained OOD) 1,083 39.89% 60.85% 66.67% 0.4382 0.0522 +26.78 pp
hard-v1/test (untrained OOD) 1,088 42.56% 58.18% 66.82% 0.4350 0.0750 +24.26 pp

2. Zero-Waste Early Exit & Real CUDA Latency (v7/test, 60 questions, avg 223 tokens, RTX PRO 6000 Blackwell)

Headline numbers are median (p50) latency to match Kevโ€™s benchmark convention (mean in parens).

| Mode (Merged LoRA + torch.compile, max |ฮ”p| = 3.5e-4) | v7/test Acc | v7/test L1 Exit | hard-v1/dev L1 Exit | Median (p50) | Mean | Speedup vs. L3 (p50) | |---|---:|---:|---:|---:|---:|---:| | Fixed Loop 3 (48L) | 85.56%* | 0.0% | 0.0% | 136.8 ms | 146.3 ms | 1.00ร— | | Fixed Loop 2 (36L) | 84.93% | โ€” | โ€” | 100.2 ms | 107.0 ms | 1.37ร— | | Fixed Loop 1 (24L) | 82.29% | 100.0% | 100.0% | 65.6 ms | 70.1 ms | 2.09ร— | | Early Exit (ฯ„ = 0.90) | 85.49% | 54.2% | 8.8% | 81.4 ms | 97.4 ms | 1.68ร— | | Early Exit (ฯ„ = 0.85) | 85.35% | 60.0% | 13.9% | 79.6 ms | 95.4 ms | 1.72ร— | | Batched (bs = 8, causal_conv1d, L1 / L3) | โ€” | โ€” | โ€” | 27.9 / 61.4 ms/q | โ€” | 35.8 / 16.3 q/s |

*Re-verified on full decision-v7/test ($n=1{,}440$) after merge_and_unload() + causal_conv1d: 86.04% (1,239 / 1,440).

Limitations & Negative Results

  • Extrapolating past Loop 3: Running 4 or 5 loops zero-shot degrades accuracy (63.5% โ†’ 59.5% โ†’ 46.1% on hard-v1/dev) because Inkling-2-Mini was pre-trained with fixed num_passes = 3.
  • Raw date arithmetic: Without deterministic date_facts preprocessing, multi-way date subtraction (hard_temporal_numeric) remains harder than multi-hop or tradeoff reasoning.
  • License: Apache 2.0 (inherited from thinkingmachines/Inkling-2-Mini).
Downloads last month
-
Safetensors
Model size
26B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for eltokh7/inkling-2-mini-decision

Finetuned
(20)
this model

Space using eltokh7/inkling-2-mini-decision 1