| ---
|
| tags:
|
| - text-classification
|
| - multiple-choice
|
| - calibrated-classification
|
| - system-one
|
| ---
|
|
|
| # decision_maker
|
|
|
| `decision_maker` is a BERT-Large decision encoder. It accepts a state, a
|
| question, and a variable list of options, then returns a calibrated probability
|
| distribution over those options in one encoder pass.
|
|
|
| ## Quick start
|
|
|
| ```bash
|
| pip install -r requirements.txt
|
| python run.py
|
| ```
|
|
|
| Edit `STATE`, `QUESTION`, and `OPTIONS` at the top of `run.py`.
|
|
|
| ## Architecture
|
|
|
| - `bert-large-uncased` bidirectional base encoder
|
| - four-layer decision Transformer
|
| - `[DECIDE]` query and `[OPTION]` key dot-product scoring
|
| - global post-training temperature scaling
|
|
|
| ## Evaluation
|
|
|
| On the held-out development split used for this release, raw vs calibrated:
|
|
|
| | Metric | Raw | Calibrated |
|
| |---|---:|---:|
|
| | Accuracy | 0.699 | 0.699 |
|
| | NLL | 0.772 | 0.625 |
|
| | Brier | 0.386 | 0.359 |
|
| | ECE | 0.121 | 0.045 |
|
|
|
| ## Limitations
|
|
|
| This model has a 256-token structured input limit. Inputs that exceed that
|
| limit are not supported by the reference runner. Calibration is measured on
|
| the release validation distribution and may not transfer to every domain.
|
| Do not use it as the sole basis for high-stakes decisions.
|
|
|
