Card: document decision heads (heads/) with JevBench results
Browse files
README.md
CHANGED
|
@@ -48,6 +48,20 @@ batch 64:
|
|
| 48 |
4.53M params, trained 200k steps with the lr schedule). Contains
|
| 49 |
`model` state dict + `config`.
|
| 50 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
## Usage
|
| 52 |
|
| 53 |
```python
|
|
|
|
| 48 |
4.53M params, trained 200k steps with the lr schedule). Contains
|
| 49 |
`model` state dict + `config`.
|
| 50 |
|
| 51 |
+
## Decision heads (`heads/`)
|
| 52 |
+
|
| 53 |
+
Fine-tuned decision readouts for the JevBench decision benchmarks — full
|
| 54 |
+
harness numbers in `docs/JEVBENCH_RESULTS.md` (GitHub):
|
| 55 |
+
|
| 56 |
+
- `decision_v1.pt` — from-scratch 4.5M decision model (synthetic decision
|
| 57 |
+
corpus, 8k steps); 16–21 ms per decision (Speed axis 100).
|
| 58 |
+
- `decision_head_2b.pt` — readout head on the frozen 2B core; best official
|
| 59 |
+
configuration (easy 39.6% / orig 34.7%, ECE 0.20, p50 0.168 s).
|
| 60 |
+
- `decision_head_2b_v2.pt` — warm-started +20k-step variant (overfits: 33.3%).
|
| 61 |
+
- `decision_head_v2b_core.pt` — head on the SFT-v2b core (35.0%).
|
| 62 |
+
- `decision_head_realdata.pt` — real-data-family corpus (CLINC150+MNLI+SST-2);
|
| 63 |
+
negative result (28.5%).
|
| 64 |
+
|
| 65 |
## Usage
|
| 66 |
|
| 67 |
```python
|