PICK-150M: Unified Geometric Decision Transformer

PICK-150M is a 149M-parameter encoder decision model built upon ModernBERT (cross-encoder/ettin-reranker-150m-v1). Designed for one-pass probabilistic decision-making, agent state guardrails, API routing, and multi-question document verification, PICK produces calibrated decision probabilities across arbitrary option sets (K) and multiple questions (Q) in a single forward pass (O(1)) via Shared-State Packing—eliminating the O(K × L²) encoding cost of traditional cross-encoders and the token sampling variance of autoregressive LLMs.

• Developer: lCREEl
• Base Encoder: cross-encoder/ettin-reranker-150m-v1 (ModernBERT-base, 22 layers, 768 hidden dimension, 12 attention heads)
• Readout Geometry: Uncentered Affine Simplex Projection (uncentered head, shared readout axis v₀ ∈ ℝ⁷⁶⁸) with RMSNorm Normalization (normalized score scaling)
• Context Capacity: 7,999 packed tokens in the final checkpoint, native RoPE (θ = 160,000) and Shared-State Block-Causal Attention

---

Complete Decision Index 0.3 public evaluation — 2026-10-19

Public index: 32.09 / 100, from the complete 140,178-request public suite. scores.json reports complete: true: 140,166 successful, 12 unsupported, 0 runtime errors, 0 pending scoreable requests. The uncorrected aggregate raw_index is 49.89; neither aggregate is a simple average accuracy.

This is the public component. The leaderboard Full score, private evaluations, and the official RTX PRO 6000 latency check are pending maintainer review.

• Registration PR #233 — submitted; awaiting maintainer review/private evaluation.
• Complete results and reproduction instructions
• Machine-readable final scores
• Independent audit and file hashes
• Frozen inference implementation

The final V5 stage-2-best weights and architecture are unchanged. This run uses native probabilities with no benchmark-specific calibration or result rewriting, fp32, compilation disabled, a 7,999-token packed capacity and forward batching budget. Every option is retained; oversized branches reject the request without truncation. Prior development used related public task families and sampled benchmark feedback; no blanket unseen-data claim is made.

The user ran the evaluation on a free Colab Tesla T4, resumed across three sessions with the same inference fingerprint. Submitted-run latency: median 35.5 ms, mean 160.9 ms, p95 465.3 ms. These are T4 timings, not the maintainers' official hardware measurement.

Full-run benchmark results

The following are native benchmark metrics on answered cases, before index coverage/chance adjustment. All scoreable requests have a recorded outcome. Six additional benchmarks are reported but do not contribute to the index.

Benchmark	Metric	Native score	Scoreable requests	Unsupported	In index
BFCL	case exact accuracy	73.67%	1,694	0	Yes
ToolRet	nDCG@10	54.62%	685	5	Yes
API-Bank	accuracy	41.34%	508	0	Yes
BANKING77	macro-F1	91.72%	3,080	0	Yes
CLINC150+OOS	macro-F1	91.09%	5,500	0	Yes
RouterBench	selected quality (quality objective)	52.03%	10,000	0	No
Home appliance simulator	case exact accuracy	94.32%	88	0	Yes
SGD/SGD-X	macro-F1	52.86%	2,500	0	No
ContractNLI	macro-F1	77.85%	123	2	Yes
ANLI	macro-F1	45.69%	3,200	0	Yes
BPoMP	accuracy	53.52%	5,000	0	Yes
Humicroedit	accuracy	52.13%	2,628	0	Yes
POP909-CL	accuracy	1.40%	2,000	4	Yes
cfcolor	accuracy	45.62%	5,000	0	Yes
MMLU	accuracy	34.11%	14,033	0	No
GPQA Diamond	accuracy	23.98%	196	0	Yes
ARC-Easy	accuracy	64.18%	2,376	0	No
ARC-Challenge	accuracy	44.20%	1,172	0	No
WinoGrande	accuracy	56.27%	1,267	0	Yes
HellaSwag	accuracy	44.53%	10,042	0	Yes
ChessBench	accuracy	7.62%	5,000	0	Yes
MuSR	accuracy	55.05%	752	0	Yes
SATA-Bench	case exact accuracy	0.18%	1,650	0	Yes
SimpleBench	accuracy	30.00%	10	0	No
BRIGHT	nDCG@10	27.19%	220	0	Yes
Amazon ESCI	macro-F1	32.57%	5,000	0	Yes
ACOS	per-review F1	22.66%	1,565	0	Yes
FinEntity	macro-F1	89.92%	979	0	Yes
iSarcasmEval	Sarcasm F1 · track A, English	14.08%	4,600	0	Yes
VAST	macro-F1	74.02%	3,006	0	Yes
NLI4CT	macro-F1	58.95%	5,500	0	Yes
CRUXEval	accuracy	41.40%	570	0	Yes
CLadder	accuracy	67.50%	5,000	0	Yes
HLE	accuracy	13.00%	501	1	Yes
Habermas Machine	accuracy	49.40%	1,676	0	Yes
GSM8K	accuracy	17.89%	2,638	0	Yes
PhishNChips phishing decisions	accuracy	49.10%	2,000	0	Yes
MMLU-Pro	accuracy	13.67%	12,032	0	Yes
BBH fixed-option tasks	accuracy	34.92%	5,507	0	Yes
RAGTruth response-level hallucination	F1 on hallucinated class	65.69%	2,700	0	Yes
HoVer claim verification	accuracy	71.30%	4,000	0	Yes
When2Call MCQ	accuracy	67.25%	3,652	0	Yes
New Yorker caption matching	accuracy	29.73%	528	0	Yes

The historical development samples below use fewer cases and optional postprocessing; their 68.84% mean is a different statistic from the full public index above.

---

Historical Development Evaluation (1,873 Requests)

PICK-150M was evaluated across 16 standardized decision benchmarks spanning agent tool calling, intent routing, clinical/legal document verification, and causal reasoning (sample=100 per benchmark, seed=42, max_length=7999), alongside unseen held-out generalization suites:

🛠️ Agent Execution, State Control & Tool Selection

Benchmark Task	Domain / Task Type	Metric	Score (Raw)	Score (Calibrated)	Context Support
Home Appliance Simulator	Interactive Device & Policy State Control	Exact Match	94.3%	94.3%	Full 8K
BFCL (Berkeley Function Calling)	Complex Tool Schema & Function Call Selection	Exact Match	81.0%	81.0%	Full 8K
When2Call MCQ	Tool Invocation Timing vs. Clarification Need	Accuracy	68.0%	74.0%	Full 8K
ToolRet	Multi-Candidate Tool Retrieval	nDCG@10	52.8%	52.8%	Full 8K
API-Bank	Live Tool & Multi-Step API Invocation	Accuracy	36.0%	36.0%	Full 8K

🧭 Domain Intent & Entity Routing

Benchmark Task	Domain / Task Type	Metric	Score (Raw)	Score (Calibrated)	Context Support
FinEntity	Financial Entity Sentiment & Stance Decision	Macro-F1	93.4%	94.7%	Full 8K
CLINC150+OOS	Multi-Domain Routing (150 classes + Out-Of-Scope)	Macro-F1	90.8%	90.8%	Full 8K
BANKING77	Fine-Grained Banking Intent (77 classes)	Macro-F1	89.0%	89.0%	Full 8K
VAST	Contextual Stance & Topic Evaluation	Macro-F1	81.0%	81.0%	Full 8K

🧠 Document Verification, NLI & Causal Reasoning

Benchmark Task	Domain / Task Type	Metric	Score (Raw)	Score (Calibrated)	Notes
ContractNLI	Legal Contract Clause Verification	Macro-F1	77.81%	77.31%	Full-contract + clause reasoning
RAGTruth	LLM Output Hallucination Detection	Class-F1	72.00%	70.27%	Response-level grounding check
HoVer	Multi-Hop Evidence Claim Verification	Accuracy	71.00%	71.00%	Cross-document fact verification
NLI4CT	Clinical Trial Protocol Inference	Macro-F1	69.85%	67.91%	Medical protocol reasoning
CLadder	Formal Causal Graph & Rung 2/3 Reasoning	Accuracy	61.00%	62.00%	Interventional & counterfactual
HellaSwag	Commonsense Situation Completion	Accuracy	44.00%	44.00%	4-choice situational completion
ACOS	Quadruple Aspect-Category Sentiment	Per-Review F1	19.56%	33.20%	Multi-quad aspect extraction
Overall Suite Mean	16-Benchmark Standardized Mean	Mean	68.84%	69.83%	1,873 total requests (Cal. Peak: 70.09%)

🎯 Held-Out Zero-Shot Generalization (Unseen Datasets)

Note: In Decision Index, accuracy and Brier score are computed at the question level (Answered Questions), as each request state contains multiple packed questions.

Evaluation Suite	Completed Requests	Evaluated Questions	Accuracy (Hits / N)	Brier Score
All Heldout	200 / 200	440	72.05% (317 / 440)	0.3892
JevAlt-en (General Decisions)	145 / 145	290	82.07% (238 / 290)	0.2541
Fast-Decisions (Domain Routing)	55 / 55	150	52.67% (79 / 150)	0.6502

---

🔬 Architecture: Shared-State Packing & Uncentered Simplex Head

Traditional cross-encoders encode each (context, question, option_k) tuple independently (O(K × L²) complexity), while centered readout heads suffer from gradient saturation when candidate set sizes (K ∈ [2, 151]) or prior distributions vary across tasks.

PICK-150M combines Shared-State Block-Causal Packing with an Uncentered Affine Simplex Geometry:

1. Shared-State Multi-Question Packing: A single document or environment state S is encoded once; multiple questions Q₁, ..., Q_M and their respective option spans C_m attend to S within a single packed sequence using structured block-causal masks, while preventing cross-contamination between distinct options.
2. Universal Readout Axis (v₀): Candidate representations are RMSNorm-normalized (õ_k = RMSNorm(o_k)) and projected onto a single shared readout direction v₀ ∈ ℝ⁷⁶⁸ with learnable temperature scaling (log_scale), preserving permutation equivariance across arbitrary K.
3. Calibrated ce-brier Objective: Jointly optimizing Cross-Entropy and Brier Score (L_CE + 0.5 · L_Brier) encourages well-calibrated decision probabilities rather than overconfident logit margins.

---

🧭 Multi-Stage Training Curriculum

Because all tasks share a single readout vector v₀, naive fine-tuning on narrow domains rotates the readout axis and degrades broad zero-shot routing. PICK-150M is trained through a disciplined multi-stage curriculum:

Phase I: Foundation Representation & Context Extension

Stage	Name	Data / Tasks	Steps	Max Length	Objective
Stage 0	AutoNCE Pretraining	287,965 multi-domain text episodes (1 pass)	8,999	1,024	Self-supervised continuation selection to align the shared readout axis v₀
Stage 1	Curated Short SFT	1,180,488 curated multi-task decisions	12,000	2,048	Multi-task supervised calibration across 45 decision archetypes
Stage 2	Full-Domain SFT	Balanced multi-domain mix (Intent, Reasoning, Multi-lingual)	12,000	2,048	Hard negative training with class-balanced cross-entropy
Stage 3	Long-Context Extension	Multi-hop contracts, clinical trials, full docs	3,000	8,192	Context extension to 8,192 tokens with RoPE frequency interpolation

Phase II: Auto-Calibrated Refinement & Generalization Polish

Stage	Name	Data / Tasks	Steps	Max Length	Objective
Stage 4	Prior Refinement (V3)	9 surgical buckets + 25% anchor replay	400	2,560	Targeted prior balancing on hard benchmarks
Stage 5	Document Polisher (V4)	Long-context legal clauses, tool routing, full documents	200	6,144	Resolves 8K document grounding & boundary consistency
Stage 6	Generalization Polish (V5)	JSON email schemas, agent traces, MASSIVE hierarchy, Fast routing	350	2,560	Structural format alignment & negative constraint reasoning
Stage 7	6K Polisher (V5)	Full multi-question long context + anchor replay	150	6,144	Final long-context robustness & decision calibration

---

💻 Quickstart Inference

The final V5 is a custom decision model. Use the frozen inference wrapper below; the repository does not provide a Transformers AutoModel remote-code mapping.

pip install https://huggingface.co/lCREEl/pick-150m/resolve/main/decision-index/pick_v5_decision_index-0.1.0-py3-none-any.whl

from pick_v5.engine import PickV5Engine
engine = PickV5Engine(device="cuda")  # or cpu / mps
response, _ = engine("The color is red.", {
    "color": {"type": "choice", "instructions": "Which color is named?",
              "criteria": {"red": "red", "blue": "blue"}}
})
print(response)

The wrapper pins original model revision 9980b9adcc14331a9b9d44bb936aca0f7deb221d and verifies every model/config/tokenizer hash. Its saved capacity is 7,999 packed tokens. No weights, architecture, training, or benchmark-specific calibration are changed. See evaluation wrapper documentation and source archive.

The benchmark tables above are historical development samples, including optional postprocessing, and are not a full Decision Index submission or leaderboard rank. The registration wrapper uses untouched native probabilities.

For free Colab execution, download and run decision-index/run_colab.sh. See the Colab shell guide for authentication, automatic HF backups, and continuation after a runtime disconnect.

---

📜 Citation

@software{pick150m2026,
  author = {lCREEl},
  title = {PICK-150M: Unified Geometric Decision Transformer with Shared-State Block-Causal Packing},
  year = {2026},
  url = {https://huggingface.co/lCREEl/pick-150m}
}