Hugging Face's logo

lCREEl
/
pick-150m

Text Classification
Transformers
Safetensors
English
modernbert
feature-extraction
decision-model
decision-transformer
ettin
cross-encoder
zero-shot-classification
classification
calibrated-probabilities
routing
multiple-choice
tool-calling
agent-decision
text-embeddings-inference
Model card Files
xet
 Community
lCREEl commited on
Commit
9980b9a
·
verified ·
1 Parent(s): c6d7c49

docs: clarify Evaluated Questions (440 = 290 JevAlt + 150 Fast) and exact calibrated metrics

Browse files
Files changed (1) hide show
  1. README.md +14 -12
README.md CHANGED
@@ -60,22 +60,24 @@ PICK-150M was evaluated across 16 standardized decision benchmarks spanning agen
60
  ### 🧠 Document Verification, NLI & Causal Reasoning
61
  | Benchmark Task | Domain / Task Type | Metric | Score (Raw) | Score (Calibrated) | Notes |
62
  | :--- | :--- | :---: | :---: | :---: | :--- |
63
- | **ContractNLI** | Legal Contract Clause Verification | Macro-F1 | **77.8%** | **77.8%** | Full-contract + clause reasoning |
64
- | **RAGTruth** | LLM Output Hallucination Detection | Class-F1 | **72.0%** | **72.0%** | Response-level grounding check |
65
- | **HoVer** | Multi-Hop Evidence Claim Verification | Accuracy | **71.0%** | **71.0%** | Cross-document fact verification |
66
- | **NLI4CT** | Clinical Trial Protocol Inference | Macro-F1 | **69.9%** | **69.9%** | Medical protocol reasoning |
67
- | **CLadder** | Formal Causal Graph & Rung 2/3 Reasoning | Accuracy | **61.0%** | **62.0%** | Interventional & counterfactual |
68
- | **HellaSwag** | Commonsense Situation Completion | Accuracy | **44.0%** | **44.0%** | 4-choice situational completion |
69
- | **ACOS** | Quadruple Aspect-Category Sentiment | Per-Review F1 | **19.6%** | **33.2%** | Multi-quad aspect extraction |
70
- | **Overall Suite Mean** | **16-Benchmark Standardized Mean** | **Mean** | **68.8%** | **69.8%** | **1,873 total requests** |
71
 
72
  ### 🎯 Held-Out Zero-Shot Generalization (Unseen Datasets)
73
 
74
- | Evaluation Suite | Completed Requests | Completion Rate | Accuracy | Brier Score |
 
 
75
  | :--- | :---: | :---: | :---: | :---: |
76
- | **All Heldout** | 200 / 200 | **100.0%** | **72.1%** | **0.3892** |
77
- | **JevAlt-en (General Decisions)** | 145 / 145 | **100.0%** | **82.1%** | **0.2541** |
78
- | **Fast-Decisions (Domain Routing)** | 55 / 55 | **100.0%** | **52.7%** | **0.6502** |
79
 
80
  ---
81
 
 
60
  ### 🧠 Document Verification, NLI & Causal Reasoning
61
  | Benchmark Task | Domain / Task Type | Metric | Score (Raw) | Score (Calibrated) | Notes |
62
  | :--- | :--- | :---: | :---: | :---: | :--- |
63
+ | **ContractNLI** | Legal Contract Clause Verification | Macro-F1 | **77.81%** | **77.31%** | Full-contract + clause reasoning |
64
+ | **RAGTruth** | LLM Output Hallucination Detection | Class-F1 | **72.00%** | **70.27%** | Response-level grounding check |
65
+ | **HoVer** | Multi-Hop Evidence Claim Verification | Accuracy | **71.00%** | **71.00%** | Cross-document fact verification |
66
+ | **NLI4CT** | Clinical Trial Protocol Inference | Macro-F1 | **69.85%** | **67.91%** | Medical protocol reasoning |
67
+ | **CLadder** | Formal Causal Graph & Rung 2/3 Reasoning | Accuracy | **61.00%** | **62.00%** | Interventional & counterfactual |
68
+ | **HellaSwag** | Commonsense Situation Completion | Accuracy | **44.00%** | **44.00%** | 4-choice situational completion |
69
+ | **ACOS** | Quadruple Aspect-Category Sentiment | Per-Review F1 | **19.56%** | **33.20%** | Multi-quad aspect extraction |
70
+ | **Overall Suite Mean** | **16-Benchmark Standardized Mean** | **Mean** | **68.84%** | **69.83%** | **1,873 total requests (Cal. Peak: 70.09%)** |
71
 
72
  ### 🎯 Held-Out Zero-Shot Generalization (Unseen Datasets)
73
 
74
+ *Note: In Decision Index, accuracy and Brier score are computed at the **question level** (`Answered Questions`), as each request state contains multiple packed questions.*
75
+
76
+ | Evaluation Suite | Completed Requests | Evaluated Questions | Accuracy (Hits / N) | Brier Score |
77
  | :--- | :---: | :---: | :---: | :---: |
78
+ | **All Heldout** | 200 / 200 | **440** | **72.05%** (317 / 440) | **0.3892** |
79
+ | **JevAlt-en (General Decisions)** | 145 / 145 | **290** | **82.07%** (238 / 290) | **0.2541** |
80
+ | **Fast-Decisions (Domain Routing)** | 55 / 55 | **150** | **52.67%** (79 / 150) | **0.6502** |
81
 
82
  ---
83