Instructions to use eltokh7/inkling-2-mini-decision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use eltokh7/inkling-2-mini-decision with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="eltokh7/inkling-2-mini-decision")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("eltokh7/inkling-2-mini-decision") model = AutoModel.from_pretrained("eltokh7/inkling-2-mini-decision", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Inkling-2-Mini Decision (eltokh7/inkling-2-mini-decision)
An open-weights System 1.5 adaptive-depth decision model built on thinkingmachines/Inkling-2-Mini (26.4B total / 3.4Bโ5.4B active MoE, 24 physical layers with a 12-layer looped core).
Instead of generating prose or JSON tokens, it evaluates a state and typed questions (choice, noul, score) in a single forward pass and dynamically decides how many internal loops (1, 2, or 3 = 24, 36, or 48 logical layers) to spend before answering โ with zero reasoning tokens.
Key Results
- Routine decisions (
decision-v7/test, $n=1{,}440$): 85.56% accuracy, 0.1944 Brier, 0.0235 ECE โ right alongside 27B-denseKev-27B(86.3%) at ~5ร lower active prefill FLOPs. - Difficulty-routed early exit (
ฯ = 0.90): 54.2% of routinedecision-v7questions exit right after Loop 1 (only 8.8% of hard OODhard-v1questions do โ 76.8% automatically route to Loop 3). - Real GPU wall-clock speed (
merge_and_unload+torch.compile, RTX PRO 6000 Blackwell, median headline / mean in parens):- Batch-1 Early Exit (
ฯ = 0.90): 81.4 ms median (97.4 msmean) โ 1.68ร faster median than compiled Loop 3 (136.8 msmedian /146.3 msmean) with โ0.07 pp accuracy difference (85.49%vs.85.56%). - Batch-1 Early Exit (
ฯ = 0.85): 79.6 ms median (95.4 msmean) โ 1.72ร faster median. - Per-pass medians (batch-1): 65.6 ms (Loop 1) / 100.2 ms (Loop 2) / 136.8 ms (Loop 3).
- Batched throughput (
bs = 8,causal_conv1d): 27.9 ms/q (35.8 q/s) at Loop 1 and 61.4 ms/q (16.3 q/s) at Loop 3.
- Batch-1 Early Exit (
- Latent reasoning on untrained OOD hard tasks (
hard-v1): Going from Loop 1 โ Loop 3 adds +26.8 pp onhard-v1/dev(39.9% โ 66.7%) and +24.3 pp onhard-v1/test(42.6% โ 66.8%), beating 9B-denseKev-9B v1(58.2%/58.4%) by +8.4 pp with 5.4B active parameters.
Architecture
- Frozen 26.4B MoE + Attention LoRA (
r = 32): LoRA adapters (17.5Mparams) attached toself_attn.{q,k,v,o}_proj.{linear,gate_proj};lm_headandvision_towerdropped for text decisioning. - Zero-Waste Layer-17 Halting Gate (
CoreGateHead,1.2Mparams): After Loop 1 or Loop 2, a lightweight bilinear probe (~0.05 ms) reads the Layer-17 core output before running exit layers18โ23. If calibrated confidence clearsthreshold, it skips the remaining loops and runs exit layers once. - Two-Stage
JointSchemaHead(6.0Mparams):- Stage 1 (Evidence Routing):
<decide>and every</opt>token cross-attend to 8 summary tokens of the input state. - Stage 2 (Cross-Option Attention): Options and
<decide>self-attend to compare directly before bilinear pointer scoring.
- Stage 1 (Evidence Routing):
- Multi-Loop Deep Supervision: Trained jointly across Loops 1, 2, and 3 (
0.2ยทL1 + 0.3ยทL2 + 0.5ยทL3) using Label-Smoothed Cross-Entropy + Brier Loss (0.5), then temperature-calibrated ondecision-v7/calibration.
Quickstart
from modeling import InklingDecisionModel
model = InklingDecisionModel.from_pretrained("eltokh7/inkling-2-mini-decision")
out = model.decide(
state="Customer email: 'My card was charged twice for order #4821 yesterday. Please refund.'",
questions={
"route": {
"type": "choice",
"instructions": "Which team should handle this ticket?",
"criteria": {
"billing": "Duplicate charges, refunds, invoices",
"tech": "Bugs, login issues, crashes",
"sales": "Pricing, upgrades, quotes",
},
},
"urgent": {
"type": "noul",
"instructions": "Does this require immediate escalation?",
},
},
threshold=0.90, # set to 1.01 to always run all 3 loops
)
print(out)
Benchmarks
1. Loop 1 vs. Loop 2 vs. Loop 3
| Split | $n$ | Loop 1 (24L) | Loop 2 (36L) | Loop 3 (48L) | L3 Brier | L3 ECE | ฮ (L3 โ L1) |
|---|---|---|---|---|---|---|---|
decision-v7/dev |
1,468 | 81.95% | 85.63% | 85.63% | 0.2056 | 0.0243 | +3.68 pp |
decision-v7/test |
1,440 | 82.29% | 84.93% | 85.56% | 0.1944 | 0.0235 | +3.27 pp |
hard-v1/dev (untrained OOD) |
1,083 | 39.89% | 60.85% | 66.67% | 0.4382 | 0.0522 | +26.78 pp |
hard-v1/test (untrained OOD) |
1,088 | 42.56% | 58.18% | 66.82% | 0.4350 | 0.0750 | +24.26 pp |
2. Zero-Waste Early Exit & Real CUDA Latency (v7/test, 60 questions, avg 223 tokens, RTX PRO 6000 Blackwell)
Headline numbers are median (p50) latency to match Kevโs benchmark convention (mean in parens).
| Mode (Merged LoRA + torch.compile, max |ฮp| = 3.5e-4) | v7/test Acc | v7/test L1 Exit | hard-v1/dev L1 Exit | Median (p50) | Mean | Speedup vs. L3 (p50) |
|---|---:|---:|---:|---:|---:|---:|
| Fixed Loop 3 (48L) | 85.56%* | 0.0% | 0.0% | 136.8 ms | 146.3 ms | 1.00ร |
| Fixed Loop 2 (36L) | 84.93% | โ | โ | 100.2 ms | 107.0 ms | 1.37ร |
| Fixed Loop 1 (24L) | 82.29% | 100.0% | 100.0% | 65.6 ms | 70.1 ms | 2.09ร |
| Early Exit (ฯ = 0.90) | 85.49% | 54.2% | 8.8% | 81.4 ms | 97.4 ms | 1.68ร |
| Early Exit (ฯ = 0.85) | 85.35% | 60.0% | 13.9% | 79.6 ms | 95.4 ms | 1.72ร |
| Batched (bs = 8, causal_conv1d, L1 / L3) | โ | โ | โ | 27.9 / 61.4 ms/q | โ | 35.8 / 16.3 q/s |
*Re-verified on full decision-v7/test ($n=1{,}440$) after merge_and_unload() + causal_conv1d: 86.04% (1,239 / 1,440).
Limitations & Negative Results
- Extrapolating past Loop 3: Running 4 or 5 loops zero-shot degrades accuracy (
63.5% โ 59.5% โ 46.1%onhard-v1/dev) becauseInkling-2-Miniwas pre-trained with fixednum_passes = 3. - Raw date arithmetic: Without deterministic
date_factspreprocessing, multi-way date subtraction (hard_temporal_numeric) remains harder than multi-hop or tradeoff reasoning. - License: Apache 2.0 (inherited from
thinkingmachines/Inkling-2-Mini).
- Downloads last month
- -
Model tree for eltokh7/inkling-2-mini-decision
Base model
thinkingmachines/Inkling-2-Mini
