File size: 1,261 Bytes
c800ccd d759515 c800ccd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | ---
tags:
- text-classification
- multiple-choice
- calibrated-classification
- system-one
---
# decision_maker
`decision_maker` is a BERT-Large decision encoder. It accepts a state, a
question, and a variable list of options, then returns a calibrated probability
distribution over those options in one encoder pass.
## Quick start
```bash
pip install -r requirements.txt
python run.py
```
Edit `STATE`, `QUESTION`, and `OPTIONS` at the top of `run.py`.
## Architecture
- `bert-large-uncased` bidirectional base encoder
- four-layer decision Transformer
- `[DECIDE]` query and `[OPTION]` key dot-product scoring
- global post-training temperature scaling
## Evaluation
On the held-out development split used for this release, raw vs calibrated:
| Metric | Raw | Calibrated |
|---|---:|---:|
| Accuracy | 0.699 | 0.699 |
| NLL | 0.772 | 0.625 |
| Brier | 0.386 | 0.359 |
| ECE | 0.121 | 0.045 |
## Limitations
This model has a 256-token structured input limit. Inputs that exceed that
limit are not supported by the reference runner. Calibration is measured on
the release validation distribution and may not transfer to every domain.
Do not use it as the sole basis for high-stakes decisions.
|