Link PICK V5 leaderboard registration PR #233

0d3d2fa verified 13 days ago

15.3 kB

---
license: cc-by-nc-4.0
language:
- en
base_model: cross-encoder/ettin-reranker-150m-v1
pipeline_tag: text-classification
library_name: transformers
tags:
- decision-model
- decision-transformer
- modernbert
- ettin
- cross-encoder
- zero-shot-classification
- classification
- calibrated-probabilities
- routing
- multiple-choice
- tool-calling
- agent-decision
datasets:
- hotchpotch/bekko-system-one-dataset-v0
- tasksource/procedural-typed-decisions
- Team-ACE/ToolACE
- glaiveai/glaive-function-calling-v2
---
# PICK-150M: Unified Geometric Decision Transformer

**PICK-150M** is a 149M-parameter encoder decision model built upon ModernBERT (`cross-encoder/ettin-reranker-150m-v1`). Designed for one-pass probabilistic decision-making, agent state guardrails, API routing, and multi-question document verification, PICK produces calibrated decision probabilities across arbitrary option sets (`K`) and multiple questions (`Q`) in a **single forward pass (`O(1)`)** via **Shared-State Packing**—eliminating the `O(K × L²)` encoding cost of traditional cross-encoders and the token sampling variance of autoregressive LLMs.

- **Developer**: [lCREEl](https://huggingface.co/lCREEl)
- **Base Encoder**: [`cross-encoder/ettin-reranker-150m-v1`](https://huggingface.co/cross-encoder/ettin-reranker-150m-v1) (ModernBERT-base, 22 layers, 768 hidden dimension, 12 attention heads)
- **Readout Geometry**: Uncentered Affine Simplex Projection (`uncentered` head, shared readout axis `v₀ ∈ ℝ⁷⁶⁸`) with RMSNorm Normalization (`normalized` score scaling)
- **Context Capacity**: 7,999 packed tokens in the final checkpoint, native RoPE (`θ = 160,000`) and Shared-State Block-Causal Attention

---

## Complete Decision Index 0.3 public evaluation — 2026-10-19

**Public index: 32.09 / 100**, from the complete **140,178-request** public suite.
`scores.json` reports `complete: true`: **140,166 successful, 12 unsupported,
0 runtime errors, 0 pending scoreable requests**. The uncorrected aggregate
`raw_index` is **49.89**; neither aggregate is a simple average accuracy.

**This is the public component. The leaderboard Full score, private evaluations,
and the official RTX PRO 6000 latency check are pending maintainer review.**

- [Registration PR #233](https://github.com/apolinario/decision-index/pull/233) — submitted; awaiting maintainer review/private evaluation.
- [Complete results and reproduction instructions](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission)
- [Machine-readable final scores](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission/blob/main/runs/pick-150m-v5/scores.json)
- [Independent audit and file hashes](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission/blob/main/final-run-audit.json)
- [Frozen inference implementation](https://huggingface.co/lCREEl/pick-150m/tree/f1635cf8868b3359c0ea78cd7d0110e7bbe6b555/decision-index)

The final V5 stage-2-best weights and architecture are unchanged. This run uses
native probabilities with no benchmark-specific calibration or result rewriting,
fp32, compilation disabled, a 7,999-token packed capacity and forward batching
budget. Every option is retained; oversized branches reject the request without
truncation. Prior development used related public task families and sampled
benchmark feedback; no blanket unseen-data claim is made.

The user ran the evaluation on a free Colab Tesla T4, resumed across three
sessions with the same inference fingerprint. Submitted-run latency: median
**35.5 ms**, mean **160.9 ms**, p95 **465.3 ms**. These are T4 timings, not the
maintainers' official hardware measurement.

### Full-run benchmark results

The following are native benchmark metrics on answered cases, before index
coverage/chance adjustment. All scoreable requests have a recorded outcome.
Six additional benchmarks are reported but do not contribute to the index.

| Benchmark | Metric | Native score | Scoreable requests | Unsupported | In index |
| :--- | :--- | ---: | ---: | ---: | :---: |
| BFCL | case exact accuracy | 73.67% | 1,694 | 0 | Yes |
| ToolRet | nDCG@10 | 54.62% | 685 | 5 | Yes |
| API-Bank | accuracy | 41.34% | 508 | 0 | Yes |
| BANKING77 | macro-F1 | 91.72% | 3,080 | 0 | Yes |
| CLINC150+OOS | macro-F1 | 91.09% | 5,500 | 0 | Yes |
| RouterBench | selected quality (quality objective) | 52.03% | 10,000 | 0 | No |
| Home appliance simulator | case exact accuracy | 94.32% | 88 | 0 | Yes |
| SGD/SGD-X | macro-F1 | 52.86% | 2,500 | 0 | No |
| ContractNLI | macro-F1 | 77.85% | 123 | 2 | Yes |
| ANLI | macro-F1 | 45.69% | 3,200 | 0 | Yes |
| BPoMP | accuracy | 53.52% | 5,000 | 0 | Yes |
| Humicroedit | accuracy | 52.13% | 2,628 | 0 | Yes |
| POP909-CL | accuracy | 1.40% | 2,000 | 4 | Yes |
| cfcolor | accuracy | 45.62% | 5,000 | 0 | Yes |
| MMLU | accuracy | 34.11% | 14,033 | 0 | No |
| GPQA Diamond | accuracy | 23.98% | 196 | 0 | Yes |
| ARC-Easy | accuracy | 64.18% | 2,376 | 0 | No |
| ARC-Challenge | accuracy | 44.20% | 1,172 | 0 | No |
| WinoGrande | accuracy | 56.27% | 1,267 | 0 | Yes |
| HellaSwag | accuracy | 44.53% | 10,042 | 0 | Yes |
| ChessBench | accuracy | 7.62% | 5,000 | 0 | Yes |
| MuSR | accuracy | 55.05% | 752 | 0 | Yes |
| SATA-Bench | case exact accuracy | 0.18% | 1,650 | 0 | Yes |
| SimpleBench | accuracy | 30.00% | 10 | 0 | No |
| BRIGHT | nDCG@10 | 27.19% | 220 | 0 | Yes |
| Amazon ESCI | macro-F1 | 32.57% | 5,000 | 0 | Yes |
| ACOS | per-review F1 | 22.66% | 1,565 | 0 | Yes |
| FinEntity | macro-F1 | 89.92% | 979 | 0 | Yes |
| iSarcasmEval | Sarcasm F1 · track A, English | 14.08% | 4,600 | 0 | Yes |
| VAST | macro-F1 | 74.02% | 3,006 | 0 | Yes |
| NLI4CT | macro-F1 | 58.95% | 5,500 | 0 | Yes |
| CRUXEval | accuracy | 41.40% | 570 | 0 | Yes |
| CLadder | accuracy | 67.50% | 5,000 | 0 | Yes |
| HLE | accuracy | 13.00% | 501 | 1 | Yes |
| Habermas Machine | accuracy | 49.40% | 1,676 | 0 | Yes |
| GSM8K | accuracy | 17.89% | 2,638 | 0 | Yes |
| PhishNChips phishing decisions | accuracy | 49.10% | 2,000 | 0 | Yes |
| MMLU-Pro | accuracy | 13.67% | 12,032 | 0 | Yes |
| BBH fixed-option tasks | accuracy | 34.92% | 5,507 | 0 | Yes |
| RAGTruth response-level hallucination | F1 on hallucinated class | 65.69% | 2,700 | 0 | Yes |
| HoVer claim verification | accuracy | 71.30% | 4,000 | 0 | Yes |
| When2Call MCQ | accuracy | 67.25% | 3,652 | 0 | Yes |
| New Yorker caption matching | accuracy | 29.73% | 528 | 0 | Yes |

The historical development samples below use fewer cases and optional postprocessing; their 68.84% mean is a different statistic from the full public index above.

---

## Historical Development Evaluation (1,873 Requests)

PICK-150M was evaluated across 16 standardized decision benchmarks spanning agent tool calling, intent routing, clinical/legal document verification, and causal reasoning (`sample=100` per benchmark, `seed=42`, `max_length=7999`), alongside unseen held-out generalization suites:

### 🛠️ Agent Execution, State Control & Tool Selection
| Benchmark Task | Domain / Task Type | Metric | Score (Raw) | Score (Calibrated) | Context Support |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **Home Appliance Simulator** | Interactive Device & Policy State Control | Exact Match | **94.3%** | **94.3%** | Full 8K |
| **BFCL (Berkeley Function Calling)** | Complex Tool Schema & Function Call Selection | Exact Match | **81.0%** | **81.0%** | Full 8K |
| **When2Call MCQ** | Tool Invocation Timing vs. Clarification Need | Accuracy | **68.0%** | **74.0%** | Full 8K |
| **ToolRet** | Multi-Candidate Tool Retrieval | nDCG@10 | **52.8%** | **52.8%** | Full 8K |
| **API-Bank** | Live Tool & Multi-Step API Invocation | Accuracy | **36.0%** | **36.0%** | Full 8K |

### 🧭 Domain Intent & Entity Routing
| Benchmark Task | Domain / Task Type | Metric | Score (Raw) | Score (Calibrated) | Context Support |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **FinEntity** | Financial Entity Sentiment & Stance Decision | Macro-F1 | **93.4%** | **94.7%** | Full 8K |
| **CLINC150+OOS** | Multi-Domain Routing (150 classes + Out-Of-Scope) | Macro-F1 | **90.8%** | **90.8%** | Full 8K |
| **BANKING77** | Fine-Grained Banking Intent (77 classes) | Macro-F1 | **89.0%** | **89.0%** | Full 8K |
| **VAST** | Contextual Stance & Topic Evaluation | Macro-F1 | **81.0%** | **81.0%** | Full 8K |

### 🧠 Document Verification, NLI & Causal Reasoning
| Benchmark Task | Domain / Task Type | Metric | Score (Raw) | Score (Calibrated) | Notes |
| :--- | :--- | :---: | :---: | :---: | :--- |
| **ContractNLI** | Legal Contract Clause Verification | Macro-F1 | **77.81%** | **77.31%** | Full-contract + clause reasoning |
| **RAGTruth** | LLM Output Hallucination Detection | Class-F1 | **72.00%** | **70.27%** | Response-level grounding check |
| **HoVer** | Multi-Hop Evidence Claim Verification | Accuracy | **71.00%** | **71.00%** | Cross-document fact verification |
| **NLI4CT** | Clinical Trial Protocol Inference | Macro-F1 | **69.85%** | **67.91%** | Medical protocol reasoning |
| **CLadder** | Formal Causal Graph & Rung 2/3 Reasoning | Accuracy | **61.00%** | **62.00%** | Interventional & counterfactual |
| **HellaSwag** | Commonsense Situation Completion | Accuracy | **44.00%** | **44.00%** | 4-choice situational completion |
| **ACOS** | Quadruple Aspect-Category Sentiment | Per-Review F1 | **19.56%** | **33.20%** | Multi-quad aspect extraction |
| **Overall Suite Mean** | **16-Benchmark Standardized Mean** | **Mean** | **68.84%** | **69.83%** | **1,873 total requests (Cal. Peak: 70.09%)** |

### 🎯 Held-Out Zero-Shot Generalization (Unseen Datasets)

*Note: In Decision Index, accuracy and Brier score are computed at the **question level** (`Answered Questions`), as each request state contains multiple packed questions.*

| Evaluation Suite | Completed Requests | Evaluated Questions | Accuracy (Hits / N) | Brier Score |
| :--- | :---: | :---: | :---: | :---: |
| **All Heldout** | 200 / 200 | **440** | **72.05%** (317 / 440) | **0.3892** |
| **JevAlt-en (General Decisions)** | 145 / 145 | **290** | **82.07%** (238 / 290) | **0.2541** |
| **Fast-Decisions (Domain Routing)** | 55 / 55 | **150** | **52.67%** (79 / 150) | **0.6502** |

---

## 🔬 Architecture: Shared-State Packing & Uncentered Simplex Head

Traditional cross-encoders encode each `(context, question, option_k)` tuple independently (`O(K × L²)` complexity), while centered readout heads suffer from gradient saturation when candidate set sizes (`K ∈ [2, 151]`) or prior distributions vary across tasks.

PICK-150M combines **Shared-State Block-Causal Packing** with an **Uncentered Affine Simplex Geometry**:
1. **Shared-State Multi-Question Packing**: A single document or environment state `S` is encoded once; multiple questions `Q₁, ..., Q_M` and their respective option spans `C_m` attend to `S` within a single packed sequence using structured block-causal masks, while preventing cross-contamination between distinct options.
2. **Universal Readout Axis (`v₀`)**: Candidate representations are RMSNorm-normalized (`õ_k = RMSNorm(o_k)`) and projected onto a single shared readout direction `v₀ ∈ ℝ⁷⁶⁸` with learnable temperature scaling (`log_scale`), preserving permutation equivariance across arbitrary `K`.
3. **Calibrated `ce-brier` Objective**: Jointly optimizing Cross-Entropy and Brier Score (`L_CE + 0.5 · L_Brier`) encourages well-calibrated decision probabilities rather than overconfident logit margins.

---

## 🧭 Multi-Stage Training Curriculum

Because all tasks share a single readout vector `v₀`, naive fine-tuning on narrow domains rotates the readout axis and degrades broad zero-shot routing. PICK-150M is trained through a disciplined multi-stage curriculum:

### Phase I: Foundation Representation & Context Extension
| Stage | Name | Data / Tasks | Steps | Max Length | Objective |
| :---: | :--- | :--- | :---: | :---: | :--- |
| **Stage 0** | **AutoNCE Pretraining** | 287,965 multi-domain text episodes (1 pass) | 8,999 | 1,024 | Self-supervised continuation selection to align the shared readout axis `v₀` |
| **Stage 1** | **Curated Short SFT** | 1,180,488 curated multi-task decisions | 12,000 | 2,048 | Multi-task supervised calibration across 45 decision archetypes |
| **Stage 2** | **Full-Domain SFT** | Balanced multi-domain mix (Intent, Reasoning, Multi-lingual) | 12,000 | 2,048 | Hard negative training with class-balanced cross-entropy |
| **Stage 3** | **Long-Context Extension** | Multi-hop contracts, clinical trials, full docs | 3,000 | 8,192 | Context extension to 8,192 tokens with RoPE frequency interpolation |

### Phase II: Auto-Calibrated Refinement & Generalization Polish
| Stage | Name | Data / Tasks | Steps | Max Length | Objective |
| :---: | :--- | :--- | :---: | :---: | :--- |
| **Stage 4** | **Prior Refinement (V3)** | 9 surgical buckets + 25% anchor replay | 400 | 2,560 | Targeted prior balancing on hard benchmarks |
| **Stage 5** | **Document Polisher (V4)** | Long-context legal clauses, tool routing, full documents | 200 | 6,144 | Resolves 8K document grounding & boundary consistency |
| **Stage 6** | **Generalization Polish (V5)** | JSON email schemas, agent traces, MASSIVE hierarchy, Fast routing | 350 | 2,560 | Structural format alignment & negative constraint reasoning |
| **Stage 7** | **6K Polisher (V5)** | Full multi-question long context + anchor replay | 150 | 6,144 | Final long-context robustness & decision calibration |

---

## 💻 Quickstart Inference

The final V5 is a custom decision model. Use the frozen inference wrapper below;
the repository does not provide a Transformers AutoModel remote-code mapping.

```sh
pip install https://huggingface.co/lCREEl/pick-150m/resolve/main/decision-index/pick_v5_decision_index-0.1.0-py3-none-any.whl
```

```python
from pick_v5.engine import PickV5Engine
engine = PickV5Engine(device="cuda")  # or cpu / mps
response, _ = engine("The color is red.", {
    "color": {"type": "choice", "instructions": "Which color is named?",
              "criteria": {"red": "red", "blue": "blue"}}
})
print(response)
```

The wrapper pins original model revision `9980b9adcc14331a9b9d44bb936aca0f7deb221d`
and verifies every model/config/tokenizer hash. Its saved capacity is **7,999 packed
tokens**. No weights, architecture, training, or benchmark-specific calibration
are changed. See [evaluation wrapper documentation](decision-index/README.md)
and [source archive](decision-index/pick-v5-inference-source.tar.gz).

The benchmark tables above are historical development samples, including optional
postprocessing, and are not a full Decision Index submission or leaderboard rank.
The registration wrapper uses untouched native probabilities.

For free Colab execution, download and run `decision-index/run_colab.sh`.
See the [Colab shell guide](decision-index/COLAB_실행방법.md) for authentication,
automatic HF backups, and continuation after a runtime disconnect.

---

## 📜 Citation

```bibtex
@software{pick150m2026,
  author = {lCREEl},
  title = {PICK-150M: Unified Geometric Decision Transformer with Shared-State Block-Causal Packing},
  year = {2026},
  url = {https://huggingface.co/lCREEl/pick-150m}
}
```

	---
	license: cc-by-nc-4.0
	language:
	- en
	base_model: cross-encoder/ettin-reranker-150m-v1
	pipeline_tag: text-classification
	library_name: transformers
	tags:
	- decision-model
	- decision-transformer
	- modernbert
	- ettin
	- cross-encoder
	- zero-shot-classification
	- classification
	- calibrated-probabilities
	- routing
	- multiple-choice
	- tool-calling
	- agent-decision
	datasets:
	- hotchpotch/bekko-system-one-dataset-v0
	- tasksource/procedural-typed-decisions
	- Team-ACE/ToolACE
	- glaiveai/glaive-function-calling-v2
	---

	# PICK-150M: Unified Geometric Decision Transformer

	PICK-150M is a 149M-parameter encoder decision model built upon ModernBERT (`cross-encoder/ettin-reranker-150m-v1`). Designed for one-pass probabilistic decision-making, agent state guardrails, API routing, and multi-question document verification, PICK produces calibrated decision probabilities across arbitrary option sets (`K`) and multiple questions (`Q`) in a single forward pass (`O(1)`) via Shared-State Packing—eliminating the `O(K × L²)` encoding cost of traditional cross-encoders and the token sampling variance of autoregressive LLMs.

	- Developer: [lCREEl](https://huggingface.co/lCREEl)
	- Base Encoder: [`cross-encoder/ettin-reranker-150m-v1`](https://huggingface.co/cross-encoder/ettin-reranker-150m-v1) (ModernBERT-base, 22 layers, 768 hidden dimension, 12 attention heads)
	- Readout Geometry: Uncentered Affine Simplex Projection (`uncentered` head, shared readout axis `v₀ ∈ ℝ⁷⁶⁸`) with RMSNorm Normalization (`normalized` score scaling)
	- Context Capacity: 7,999 packed tokens in the final checkpoint, native RoPE (`θ = 160,000`) and Shared-State Block-Causal Attention

	---

	## Complete Decision Index 0.3 public evaluation — 2026-10-19

	Public index: 32.09 / 100, from the complete 140,178-request public suite.
	`scores.json` reports `complete: true`: **140,166 successful, 12 unsupported,
	0 runtime errors, 0 pending scoreable requests**. The uncorrected aggregate
	`raw_index` is 49.89; neither aggregate is a simple average accuracy.

	**This is the public component. The leaderboard Full score, private evaluations,
	and the official RTX PRO 6000 latency check are pending maintainer review.**

	- [Registration PR #233](https://github.com/apolinario/decision-index/pull/233) — submitted; awaiting maintainer review/private evaluation.
	- [Complete results and reproduction instructions](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission)
	- [Machine-readable final scores](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission/blob/main/runs/pick-150m-v5/scores.json)
	- [Independent audit and file hashes](https://huggingface.co/datasets/lCREEl/pick-v5-decision-index-submission/blob/main/final-run-audit.json)
	- [Frozen inference implementation](https://huggingface.co/lCREEl/pick-150m/tree/f1635cf8868b3359c0ea78cd7d0110e7bbe6b555/decision-index)

	The final V5 stage-2-best weights and architecture are unchanged. This run uses
	native probabilities with no benchmark-specific calibration or result rewriting,
	fp32, compilation disabled, a 7,999-token packed capacity and forward batching
	budget. Every option is retained; oversized branches reject the request without
	truncation. Prior development used related public task families and sampled
	benchmark feedback; no blanket unseen-data claim is made.

	The user ran the evaluation on a free Colab Tesla T4, resumed across three
	sessions with the same inference fingerprint. Submitted-run latency: median
	35.5 ms, mean 160.9 ms, p95 465.3 ms. These are T4 timings, not the
	maintainers' official hardware measurement.

	### Full-run benchmark results

	The following are native benchmark metrics on answered cases, before index
	coverage/chance adjustment. All scoreable requests have a recorded outcome.
	Six additional benchmarks are reported but do not contribute to the index.

	\| Benchmark \| Metric \| Native score \| Scoreable requests \| Unsupported \| In index \|
	\| :--- \| :--- \| ---: \| ---: \| ---: \| :---: \|
	\| BFCL \| case exact accuracy \| 73.67% \| 1,694 \| 0 \| Yes \|
	\| ToolRet \| nDCG@10 \| 54.62% \| 685 \| 5 \| Yes \|
	\| API-Bank \| accuracy \| 41.34% \| 508 \| 0 \| Yes \|
	\| BANKING77 \| macro-F1 \| 91.72% \| 3,080 \| 0 \| Yes \|
	\| CLINC150+OOS \| macro-F1 \| 91.09% \| 5,500 \| 0 \| Yes \|
	\| RouterBench \| selected quality (quality objective) \| 52.03% \| 10,000 \| 0 \| No \|
	\| Home appliance simulator \| case exact accuracy \| 94.32% \| 88 \| 0 \| Yes \|
	\| SGD/SGD-X \| macro-F1 \| 52.86% \| 2,500 \| 0 \| No \|
	\| ContractNLI \| macro-F1 \| 77.85% \| 123 \| 2 \| Yes \|
	\| ANLI \| macro-F1 \| 45.69% \| 3,200 \| 0 \| Yes \|
	\| BPoMP \| accuracy \| 53.52% \| 5,000 \| 0 \| Yes \|
	\| Humicroedit \| accuracy \| 52.13% \| 2,628 \| 0 \| Yes \|
	\| POP909-CL \| accuracy \| 1.40% \| 2,000 \| 4 \| Yes \|
	\| cfcolor \| accuracy \| 45.62% \| 5,000 \| 0 \| Yes \|
	\| MMLU \| accuracy \| 34.11% \| 14,033 \| 0 \| No \|
	\| GPQA Diamond \| accuracy \| 23.98% \| 196 \| 0 \| Yes \|
	\| ARC-Easy \| accuracy \| 64.18% \| 2,376 \| 0 \| No \|
	\| ARC-Challenge \| accuracy \| 44.20% \| 1,172 \| 0 \| No \|
	\| WinoGrande \| accuracy \| 56.27% \| 1,267 \| 0 \| Yes \|
	\| HellaSwag \| accuracy \| 44.53% \| 10,042 \| 0 \| Yes \|
	\| ChessBench \| accuracy \| 7.62% \| 5,000 \| 0 \| Yes \|
	\| MuSR \| accuracy \| 55.05% \| 752 \| 0 \| Yes \|
	\| SATA-Bench \| case exact accuracy \| 0.18% \| 1,650 \| 0 \| Yes \|
	\| SimpleBench \| accuracy \| 30.00% \| 10 \| 0 \| No \|
	\| BRIGHT \| nDCG@10 \| 27.19% \| 220 \| 0 \| Yes \|
	\| Amazon ESCI \| macro-F1 \| 32.57% \| 5,000 \| 0 \| Yes \|
	\| ACOS \| per-review F1 \| 22.66% \| 1,565 \| 0 \| Yes \|
	\| FinEntity \| macro-F1 \| 89.92% \| 979 \| 0 \| Yes \|
	\| iSarcasmEval \| Sarcasm F1 · track A, English \| 14.08% \| 4,600 \| 0 \| Yes \|
	\| VAST \| macro-F1 \| 74.02% \| 3,006 \| 0 \| Yes \|
	\| NLI4CT \| macro-F1 \| 58.95% \| 5,500 \| 0 \| Yes \|
	\| CRUXEval \| accuracy \| 41.40% \| 570 \| 0 \| Yes \|
	\| CLadder \| accuracy \| 67.50% \| 5,000 \| 0 \| Yes \|
	\| HLE \| accuracy \| 13.00% \| 501 \| 1 \| Yes \|
	\| Habermas Machine \| accuracy \| 49.40% \| 1,676 \| 0 \| Yes \|
	\| GSM8K \| accuracy \| 17.89% \| 2,638 \| 0 \| Yes \|
	\| PhishNChips phishing decisions \| accuracy \| 49.10% \| 2,000 \| 0 \| Yes \|
	\| MMLU-Pro \| accuracy \| 13.67% \| 12,032 \| 0 \| Yes \|
	\| BBH fixed-option tasks \| accuracy \| 34.92% \| 5,507 \| 0 \| Yes \|
	\| RAGTruth response-level hallucination \| F1 on hallucinated class \| 65.69% \| 2,700 \| 0 \| Yes \|
	\| HoVer claim verification \| accuracy \| 71.30% \| 4,000 \| 0 \| Yes \|
	\| When2Call MCQ \| accuracy \| 67.25% \| 3,652 \| 0 \| Yes \|
	\| New Yorker caption matching \| accuracy \| 29.73% \| 528 \| 0 \| Yes \|

	The historical development samples below use fewer cases and optional postprocessing; their 68.84% mean is a different statistic from the full public index above.

	---

	## Historical Development Evaluation (1,873 Requests)

	PICK-150M was evaluated across 16 standardized decision benchmarks spanning agent tool calling, intent routing, clinical/legal document verification, and causal reasoning (`sample=100` per benchmark, `seed=42`, `max_length=7999`), alongside unseen held-out generalization suites:

	### 🛠️ Agent Execution, State Control & Tool Selection
	\| Benchmark Task \| Domain / Task Type \| Metric \| Score (Raw) \| Score (Calibrated) \| Context Support \|
	\| :--- \| :--- \| :---: \| :---: \| :---: \| :---: \|
	\| Home Appliance Simulator \| Interactive Device & Policy State Control \| Exact Match \| 94.3% \| 94.3% \| Full 8K \|
	\| BFCL (Berkeley Function Calling) \| Complex Tool Schema & Function Call Selection \| Exact Match \| 81.0% \| 81.0% \| Full 8K \|
	\| When2Call MCQ \| Tool Invocation Timing vs. Clarification Need \| Accuracy \| 68.0% \| 74.0% \| Full 8K \|
	\| ToolRet \| Multi-Candidate Tool Retrieval \| nDCG@10 \| 52.8% \| 52.8% \| Full 8K \|
	\| API-Bank \| Live Tool & Multi-Step API Invocation \| Accuracy \| 36.0% \| 36.0% \| Full 8K \|

	### 🧭 Domain Intent & Entity Routing
	\| Benchmark Task \| Domain / Task Type \| Metric \| Score (Raw) \| Score (Calibrated) \| Context Support \|
	\| :--- \| :--- \| :---: \| :---: \| :---: \| :---: \|
	\| FinEntity \| Financial Entity Sentiment & Stance Decision \| Macro-F1 \| 93.4% \| 94.7% \| Full 8K \|
	\| CLINC150+OOS \| Multi-Domain Routing (150 classes + Out-Of-Scope) \| Macro-F1 \| 90.8% \| 90.8% \| Full 8K \|
	\| BANKING77 \| Fine-Grained Banking Intent (77 classes) \| Macro-F1 \| 89.0% \| 89.0% \| Full 8K \|
	\| VAST \| Contextual Stance & Topic Evaluation \| Macro-F1 \| 81.0% \| 81.0% \| Full 8K \|

	### 🧠 Document Verification, NLI & Causal Reasoning
	\| Benchmark Task \| Domain / Task Type \| Metric \| Score (Raw) \| Score (Calibrated) \| Notes \|
	\| :--- \| :--- \| :---: \| :---: \| :---: \| :--- \|
	\| ContractNLI \| Legal Contract Clause Verification \| Macro-F1 \| 77.81% \| 77.31% \| Full-contract + clause reasoning \|
	\| RAGTruth \| LLM Output Hallucination Detection \| Class-F1 \| 72.00% \| 70.27% \| Response-level grounding check \|
	\| HoVer \| Multi-Hop Evidence Claim Verification \| Accuracy \| 71.00% \| 71.00% \| Cross-document fact verification \|
	\| NLI4CT \| Clinical Trial Protocol Inference \| Macro-F1 \| 69.85% \| 67.91% \| Medical protocol reasoning \|
	\| CLadder \| Formal Causal Graph & Rung 2/3 Reasoning \| Accuracy \| 61.00% \| 62.00% \| Interventional & counterfactual \|
	\| HellaSwag \| Commonsense Situation Completion \| Accuracy \| 44.00% \| 44.00% \| 4-choice situational completion \|
	\| ACOS \| Quadruple Aspect-Category Sentiment \| Per-Review F1 \| 19.56% \| 33.20% \| Multi-quad aspect extraction \|
	\| Overall Suite Mean \| 16-Benchmark Standardized Mean \| Mean \| 68.84% \| 69.83% \| 1,873 total requests (Cal. Peak: 70.09%) \|

	### 🎯 Held-Out Zero-Shot Generalization (Unseen Datasets)

	Note: In Decision Index, accuracy and Brier score are computed at the question level* (`Answered Questions`), as each request state contains multiple packed questions.*

	\| Evaluation Suite \| Completed Requests \| Evaluated Questions \| Accuracy (Hits / N) \| Brier Score \|
	\| :--- \| :---: \| :---: \| :---: \| :---: \|
	\| All Heldout \| 200 / 200 \| 440 \| 72.05% (317 / 440) \| 0.3892 \|
	\| JevAlt-en (General Decisions) \| 145 / 145 \| 290 \| 82.07% (238 / 290) \| 0.2541 \|
	\| Fast-Decisions (Domain Routing) \| 55 / 55 \| 150 \| 52.67% (79 / 150) \| 0.6502 \|

	---

	## 🔬 Architecture: Shared-State Packing & Uncentered Simplex Head

	Traditional cross-encoders encode each `(context, question, option_k)` tuple independently (`O(K × L²)` complexity), while centered readout heads suffer from gradient saturation when candidate set sizes (`K ∈ [2, 151]`) or prior distributions vary across tasks.

	PICK-150M combines Shared-State Block-Causal Packing with an Uncentered Affine Simplex Geometry:
	1. Shared-State Multi-Question Packing: A single document or environment state `S` is encoded once; multiple questions `Q₁, ..., Q_M` and their respective option spans `C_m` attend to `S` within a single packed sequence using structured block-causal masks, while preventing cross-contamination between distinct options.
	2. Universal Readout Axis (`v₀`): Candidate representations are RMSNorm-normalized (`õ_k = RMSNorm(o_k)`) and projected onto a single shared readout direction `v₀ ∈ ℝ⁷⁶⁸` with learnable temperature scaling (`log_scale`), preserving permutation equivariance across arbitrary `K`.
	3. Calibrated `ce-brier` Objective: Jointly optimizing Cross-Entropy and Brier Score (`L_CE + 0.5 · L_Brier`) encourages well-calibrated decision probabilities rather than overconfident logit margins.

	---

	## 🧭 Multi-Stage Training Curriculum

	Because all tasks share a single readout vector `v₀`, naive fine-tuning on narrow domains rotates the readout axis and degrades broad zero-shot routing. PICK-150M is trained through a disciplined multi-stage curriculum:

	### Phase I: Foundation Representation & Context Extension
	\| Stage \| Name \| Data / Tasks \| Steps \| Max Length \| Objective \|
	\| :---: \| :--- \| :--- \| :---: \| :---: \| :--- \|
	\| Stage 0 \| AutoNCE Pretraining \| 287,965 multi-domain text episodes (1 pass) \| 8,999 \| 1,024 \| Self-supervised continuation selection to align the shared readout axis `v₀` \|
	\| Stage 1 \| Curated Short SFT \| 1,180,488 curated multi-task decisions \| 12,000 \| 2,048 \| Multi-task supervised calibration across 45 decision archetypes \|
	\| Stage 2 \| Full-Domain SFT \| Balanced multi-domain mix (Intent, Reasoning, Multi-lingual) \| 12,000 \| 2,048 \| Hard negative training with class-balanced cross-entropy \|
	\| Stage 3 \| Long-Context Extension \| Multi-hop contracts, clinical trials, full docs \| 3,000 \| 8,192 \| Context extension to 8,192 tokens with RoPE frequency interpolation \|

	### Phase II: Auto-Calibrated Refinement & Generalization Polish
	\| Stage \| Name \| Data / Tasks \| Steps \| Max Length \| Objective \|
	\| :---: \| :--- \| :--- \| :---: \| :---: \| :--- \|
	\| Stage 4 \| Prior Refinement (V3) \| 9 surgical buckets + 25% anchor replay \| 400 \| 2,560 \| Targeted prior balancing on hard benchmarks \|
	\| Stage 5 \| Document Polisher (V4) \| Long-context legal clauses, tool routing, full documents \| 200 \| 6,144 \| Resolves 8K document grounding & boundary consistency \|
	\| Stage 6 \| Generalization Polish (V5) \| JSON email schemas, agent traces, MASSIVE hierarchy, Fast routing \| 350 \| 2,560 \| Structural format alignment & negative constraint reasoning \|
	\| Stage 7 \| 6K Polisher (V5) \| Full multi-question long context + anchor replay \| 150 \| 6,144 \| Final long-context robustness & decision calibration \|

	---

	## 💻 Quickstart Inference

	The final V5 is a custom decision model. Use the frozen inference wrapper below;
	the repository does not provide a Transformers AutoModel remote-code mapping.

	```sh
	pip install https://huggingface.co/lCREEl/pick-150m/resolve/main/decision-index/pick_v5_decision_index-0.1.0-py3-none-any.whl
	```

	```python
	from pick_v5.engine import PickV5Engine
	engine = PickV5Engine(device="cuda") # or cpu / mps
	response, _ = engine("The color is red.", {
	"color": {"type": "choice", "instructions": "Which color is named?",
	"criteria": {"red": "red", "blue": "blue"}}
	})
	print(response)
	```

	The wrapper pins original model revision `9980b9adcc14331a9b9d44bb936aca0f7deb221d`
	and verifies every model/config/tokenizer hash. Its saved capacity is **7,999 packed
	tokens**. No weights, architecture, training, or benchmark-specific calibration
	are changed. See [evaluation wrapper documentation](decision-index/README.md)
	and [source archive](decision-index/pick-v5-inference-source.tar.gz).

	The benchmark tables above are historical development samples, including optional
	postprocessing, and are not a full Decision Index submission or leaderboard rank.
	The registration wrapper uses untouched native probabilities.

	For free Colab execution, download and run `decision-index/run_colab.sh`.
	See the [Colab shell guide](decision-index/COLAB_실행방법.md) for authentication,
	automatic HF backups, and continuation after a runtime disconnect.

	---

	## 📜 Citation

	```bibtex
	@software{pick150m2026,
	author = {lCREEl},
	title = {PICK-150M: Unified Geometric Decision Transformer with Shared-State Block-Causal Packing},
	year = {2026},
	url = {https://huggingface.co/lCREEl/pick-150m}
	}
	```