---
license: cc-by-nc-4.0
language:
- en
base_model: cross-encoder/ettin-reranker-150m-v1
pipeline_tag: text-classification
library_name: transformers
tags:
- decision-model
- decision-transformer
- modernbert
- ettin
- cross-encoder
- zero-shot-classification
- classification
- calibrated-probabilities
- routing
- multiple-choice
- tool-calling
- agent-decision
datasets:
- hotchpotch/bekko-system-one-dataset-v0
- tasksource/procedural-typed-decisions
- Team-ACE/ToolACE
- glaiveai/glaive-function-calling-v2
---

# PICK-150M: Unified Geometric Decision Transformer

**PICK-150M** is a 149M-parameter encoder decision model built upon ModernBERT (`cross-encoder/ettin-reranker-150m-v1`). Designed for one-pass probabilistic decision-making, agent state guardrails, API routing, and multi-question document verification, PICK produces calibrated decision probabilities across arbitrary option sets (`K`) and multiple questions (`Q`) in a **single forward pass (`O(1)`)** via **Shared-State Packing**—eliminating the `O(K × L²)` encoding cost of traditional cross-encoders and the token sampling variance of autoregressive LLMs.

- **Developer**: [lCREEl](https://huggingface.co/lCREEl)
- **Base Encoder**: [`cross-encoder/ettin-reranker-150m-v1`](https://huggingface.co/cross-encoder/ettin-reranker-150m-v1) (ModernBERT-base, 22 layers, 768 hidden dimension, 12 attention heads)
- **Readout Geometry**: Uncentered Affine Simplex Projection (`uncentered` head, shared readout axis `v₀ ∈ ℝ⁷⁶⁸`) with RMSNorm Normalization (`normalized` score scaling)
- **Context Capacity**: 7,999 packed tokens in the final checkpoint, native RoPE (`θ = 160,000`) and Shared-State Block-Causal Attention

---

## Complete Decision Index 0.3 public evaluation — 2026-10-19

**Public index: 32.09 / 100**, from the complete **140,178-request** public suite.
`scores.json` reports `complete: true`: **140,166 successful, 12 unsupported,
0 runtime errors, 0 pending scoreable requests**. The uncorrected aggregate
`raw_index` is **49.89**; neither aggregate is a simple average accuracy.

**This is the public component. The leaderboard Full score, private evaluations,
and the official RTX PRO 6000 latency check are pending maintainer review.**

- [Registration PR #233](https://github.com/apolinario/decision-index/pull/233) — submitted; awaiting maintainer review/private evaluation.
- [Complete results and reproduction instructions](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission)
- [Machine-readable final scores](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission/blob/main/runs/pick-150m-v5/scores.json)
- [Independent audit and file hashes](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission/blob/main/final-run-audit.json)
- [Frozen inference implementation](https://huggingface.co/lCREEl/pick-150m/tree/f1635cf8868b3359c0ea78cd7d0110e7bbe6b555/decision-index)

The final V5 stage-2-best weights and architecture are unchanged. This run uses
native probabilities with no benchmark-specific calibration or result rewriting,
fp32, compilation disabled, a 7,999-token packed capacity and forward batching
budget. Every option is retained; oversized branches reject the request without
truncation. Prior development used related public task families and sampled
benchmark feedback; no blanket unseen-data claim is made.

The user ran the evaluation on a free Colab Tesla T4, resumed across three
sessions with the same inference fingerprint. Submitted-run latency: median
**35.5 ms**, mean **160.9 ms**, p95 **465.3 ms**. These are T4 timings, not the
maintainers' official hardware measurement.

### Full-run benchmark results

The following are native benchmark metrics on answered cases, before index
coverage/chance adjustment. All scoreable requests have a recorded outcome.
Six additional benchmarks are reported but do not contribute to the index.

| Benchmark | Metric | Native score | Scoreable requests | Unsupported | In index |
| :--- | :--- | ---: | ---: | ---: | :---: |
| BFCL | case exact accuracy | 73.67% | 1,694 | 0 | Yes |
| ToolRet | nDCG@10 | 54.62% | 685 | 5 | Yes |
| API-Bank | accuracy | 41.34% | 508 | 0 | Yes |
| BANKING77 | macro-F1 | 91.72% | 3,080 | 0 | Yes |
| CLINC150+OOS | macro-F1 | 91.09% | 5,500 | 0 | Yes |
| RouterBench | selected quality (quality objective) | 52.03% | 10,000 | 0 | No |
| Home appliance simulator | case exact accuracy | 94.32% | 88 | 0 | Yes |
| SGD/SGD-X | macro-F1 | 52.86% | 2,500 | 0 | No |
| ContractNLI | macro-F1 | 77.85% | 123 | 2 | Yes |
| ANLI | macro-F1 | 45.69% | 3,200 | 0 | Yes |
| BPoMP | accuracy | 53.52% | 5,000 | 0 | Yes |
| Humicroedit | accuracy | 52.13% | 2,628 | 0 | Yes |
| POP909-CL | accuracy | 1.40% | 2,000 | 4 | Yes |
| cfcolor | accuracy | 45.62% | 5,000 | 0 | Yes |
| MMLU | accuracy | 34.11% | 14,033 | 0 | No |
| GPQA Diamond | accuracy | 23.98% | 196 | 0 | Yes |
| ARC-Easy | accuracy | 64.18% | 2,376 | 0 | No |
| ARC-Challenge | accuracy | 44.20% | 1,172 | 0 | No |
| WinoGrande | accuracy | 56.27% | 1,267 | 0 | Yes |
| HellaSwag | accuracy | 44.53% | 10,042 | 0 | Yes |
| ChessBench | accuracy | 7.62% | 5,000 | 0 | Yes |
| MuSR | accuracy | 55.05% | 752 | 0 | Yes |
| SATA-Bench | case exact accuracy | 0.18% | 1,650 | 0 | Yes |
| SimpleBench | accuracy | 30.00% | 10 | 0 | No |
| BRIGHT | nDCG@10 | 27.19% | 220 | 0 | Yes |
| Amazon ESCI | macro-F1 | 32.57% | 5,000 | 0 | Yes |
| ACOS | per-review F1 | 22.66% | 1,565 | 0 | Yes |
| FinEntity | macro-F1 | 89.92% | 979 | 0 | Yes |
| iSarcasmEval | Sarcasm F1 · track A, English | 14.08% | 4,600 | 0 | Yes |
| VAST | macro-F1 | 74.02% | 3,006 | 0 | Yes |
| NLI4CT | macro-F1 | 58.95% | 5,500 | 0 | Yes |
| CRUXEval | accuracy | 41.40% | 570 | 0 | Yes |
| CLadder | accuracy | 67.50% | 5,000 | 0 | Yes |
| HLE | accuracy | 13.00% | 501 | 1 | Yes |
| Habermas Machine | accuracy | 49.40% | 1,676 | 0 | Yes |
| GSM8K | accuracy | 17.89% | 2,638 | 0 | Yes |
| PhishNChips phishing decisions | accuracy | 49.10% | 2,000 | 0 | Yes |
| MMLU-Pro | accuracy | 13.67% | 12,032 | 0 | Yes |
| BBH fixed-option tasks | accuracy | 34.92% | 5,507 | 0 | Yes |
| RAGTruth response-level hallucination | F1 on hallucinated class | 65.69% | 2,700 | 0 | Yes |
| HoVer claim verification | accuracy | 71.30% | 4,000 | 0 | Yes |
| When2Call MCQ | accuracy | 67.25% | 3,652 | 0 | Yes |
| New Yorker caption matching | accuracy | 29.73% | 528 | 0 | Yes |

The historical development samples below use fewer cases and optional postprocessing; their 68.84% mean is a different statistic from the full public index above.

---

## Historical Development Evaluation (1,873 Requests)

PICK-150M was evaluated across 16 standardized decision benchmarks spanning agent tool calling, intent routing, clinical/legal document verification, and causal reasoning (`sample=100` per benchmark, `seed=42`, `max_length=7999`), alongside unseen held-out generalization suites:

### 🛠️ Agent Execution, State Control & Tool Selection
| Benchmark Task | Domain / Task Type | Metric | Score (Raw) | Score (Calibrated) | Context Support |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **Home Appliance Simulator** | Interactive Device & Policy State Control | Exact Match | **94.3%** | **94.3%** | Full 8K |
| **BFCL (Berkeley Function Calling)** | Complex Tool Schema & Function Call Selection | Exact Match | **81.0%** | **81.0%** | Full 8K |
| **When2Call MCQ** | Tool Invocation Timing vs. Clarification Need | Accuracy | **68.0%** | **74.0%** | Full 8K |
| **ToolRet** | Multi-Candidate Tool Retrieval | nDCG@10 | **52.8%** | **52.8%** | Full 8K |
| **API-Bank** | Live Tool & Multi-Step API Invocation | Accuracy | **36.0%** | **36.0%** | Full 8K |

### 🧭 Domain Intent & Entity Routing
| Benchmark Task | Domain / Task Type | Metric | Score (Raw) | Score (Calibrated) | Context Support |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **FinEntity** | Financial Entity Sentiment & Stance Decision | Macro-F1 | **93.4%** | **94.7%** | Full 8K |
| **CLINC150+OOS** | Multi-Domain Routing (150 classes + Out-Of-Scope) | Macro-F1 | **90.8%** | **90.8%** | Full 8K |
| **BANKING77** | Fine-Grained Banking Intent (77 classes) | Macro-F1 | **89.0%** | **89.0%** | Full 8K |
| **VAST** | Contextual Stance & Topic Evaluation | Macro-F1 | **81.0%** | **81.0%** | Full 8K |

### 🧠 Document Verification, NLI & Causal Reasoning
| Benchmark Task | Domain / Task Type | Metric | Score (Raw) | Score (Calibrated) | Notes |
| :--- | :--- | :---: | :---: | :---: | :--- |
| **ContractNLI** | Legal Contract Clause Verification | Macro-F1 | **77.81%** | **77.31%** | Full-contract + clause reasoning |
| **RAGTruth** | LLM Output Hallucination Detection | Class-F1 | **72.00%** | **70.27%** | Response-level grounding check |
| **HoVer** | Multi-Hop Evidence Claim Verification | Accuracy | **71.00%** | **71.00%** | Cross-document fact verification |
| **NLI4CT** | Clinical Trial Protocol Inference | Macro-F1 | **69.85%** | **67.91%** | Medical protocol reasoning |
| **CLadder** | Formal Causal Graph & Rung 2/3 Reasoning | Accuracy | **61.00%** | **62.00%** | Interventional & counterfactual |
| **HellaSwag** | Commonsense Situation Completion | Accuracy | **44.00%** | **44.00%** | 4-choice situational completion |
| **ACOS** | Quadruple Aspect-Category Sentiment | Per-Review F1 | **19.56%** | **33.20%** | Multi-quad aspect extraction |
| **Overall Suite Mean** | **16-Benchmark Standardized Mean** | **Mean** | **68.84%** | **69.83%** | **1,873 total requests (Cal. Peak: 70.09%)** |

### 🎯 Held-Out Zero-Shot Generalization (Unseen Datasets)

*Note: In Decision Index, accuracy and Brier score are computed at the **question level** (`Answered Questions`), as each request state contains multiple packed questions.*

| Evaluation Suite | Completed Requests | Evaluated Questions | Accuracy (Hits / N) | Brier Score |
| :--- | :---: | :---: | :---: | :---: |
| **All Heldout** | 200 / 200 | **440** | **72.05%** (317 / 440) | **0.3892** |
| **JevAlt-en (General Decisions)** | 145 / 145 | **290** | **82.07%** (238 / 290) | **0.2541** |
| **Fast-Decisions (Domain Routing)** | 55 / 55 | **150** | **52.67%** (79 / 150) | **0.6502** |

---

## 🔬 Architecture: Shared-State Packing & Uncentered Simplex Head

Traditional cross-encoders encode each `(context, question, option_k)` tuple independently (`O(K × L²)` complexity), while centered readout heads suffer from gradient saturation when candidate set sizes (`K ∈ [2, 151]`) or prior distributions vary across tasks.

PICK-150M combines **Shared-State Block-Causal Packing** with an **Uncentered Affine Simplex Geometry**:
1. **Shared-State Multi-Question Packing**: A single document or environment state `S` is encoded once; multiple questions `Q₁, ..., Q_M` and their respective option spans `C_m` attend to `S` within a single packed sequence using structured block-causal masks, while preventing cross-contamination between distinct options.
2. **Universal Readout Axis (`v₀`)**: Candidate representations are RMSNorm-normalized (`õ_k = RMSNorm(o_k)`) and projected onto a single shared readout direction `v₀ ∈ ℝ⁷⁶⁸` with learnable temperature scaling (`log_scale`), preserving permutation equivariance across arbitrary `K`.
3. **Calibrated `ce-brier` Objective**: Jointly optimizing Cross-Entropy and Brier Score (`L_CE + 0.5 · L_Brier`) encourages well-calibrated decision probabilities rather than overconfident logit margins.

---

## 🧭 Multi-Stage Training Curriculum

Because all tasks share a single readout vector `v₀`, naive fine-tuning on narrow domains rotates the readout axis and degrades broad zero-shot routing. PICK-150M is trained through a disciplined multi-stage curriculum:

### Phase I: Foundation Representation & Context Extension
| Stage | Name | Data / Tasks | Steps | Max Length | Objective |
| :---: | :--- | :--- | :---: | :---: | :--- |
| **Stage 0** | **AutoNCE Pretraining** | 287,965 multi-domain text episodes (1 pass) | 8,999 | 1,024 | Self-supervised continuation selection to align the shared readout axis `v₀` |
| **Stage 1** | **Curated Short SFT** | 1,180,488 curated multi-task decisions | 12,000 | 2,048 | Multi-task supervised calibration across 45 decision archetypes |
| **Stage 2** | **Full-Domain SFT** | Balanced multi-domain mix (Intent, Reasoning, Multi-lingual) | 12,000 | 2,048 | Hard negative training with class-balanced cross-entropy |
| **Stage 3** | **Long-Context Extension** | Multi-hop contracts, clinical trials, full docs | 3,000 | 8,192 | Context extension to 8,192 tokens with RoPE frequency interpolation |

### Phase II: Auto-Calibrated Refinement & Generalization Polish
| Stage | Name | Data / Tasks | Steps | Max Length | Objective |
| :---: | :--- | :--- | :---: | :---: | :--- |
| **Stage 4** | **Prior Refinement (V3)** | 9 surgical buckets + 25% anchor replay | 400 | 2,560 | Targeted prior balancing on hard benchmarks |
| **Stage 5** | **Document Polisher (V4)** | Long-context legal clauses, tool routing, full documents | 200 | 6,144 | Resolves 8K document grounding & boundary consistency |
| **Stage 6** | **Generalization Polish (V5)** | JSON email schemas, agent traces, MASSIVE hierarchy, Fast routing | 350 | 2,560 | Structural format alignment & negative constraint reasoning |
| **Stage 7** | **6K Polisher (V5)** | Full multi-question long context + anchor replay | 150 | 6,144 | Final long-context robustness & decision calibration |

---

## 💻 Quickstart Inference

The final V5 is a custom decision model. Use the frozen inference wrapper below;
the repository does not provide a Transformers AutoModel remote-code mapping.

```sh
pip install https://huggingface.co/lCREEl/pick-150m/resolve/main/decision-index/pick_v5_decision_index-0.1.0-py3-none-any.whl
```

```python
from pick_v5.engine import PickV5Engine
engine = PickV5Engine(device="cuda")  # or cpu / mps
response, _ = engine("The color is red.", {
    "color": {"type": "choice", "instructions": "Which color is named?",
              "criteria": {"red": "red", "blue": "blue"}}
})
print(response)
```

The wrapper pins original model revision `9980b9adcc14331a9b9d44bb936aca0f7deb221d`
and verifies every model/config/tokenizer hash. Its saved capacity is **7,999 packed
tokens**. No weights, architecture, training, or benchmark-specific calibration
are changed. See [evaluation wrapper documentation](decision-index/README.md)
and [source archive](decision-index/pick-v5-inference-source.tar.gz).

The benchmark tables above are historical development samples, including optional
postprocessing, and are not a full Decision Index submission or leaderboard rank.
The registration wrapper uses untouched native probabilities.

For free Colab execution, download and run `decision-index/run_colab.sh`.
See the [Colab shell guide](decision-index/COLAB_실행방법.md) for authentication,
automatic HF backups, and continuation after a runtime disconnect.

---

## 📜 Citation

```bibtex
@software{pick150m2026,
  author = {lCREEl},
  title = {PICK-150M: Unified Geometric Decision Transformer with Shared-State Block-Causal Packing},
  year = {2026},
  url = {https://huggingface.co/lCREEl/pick-150m}
}
```