native temperature refit (10.5667)
Browse files- MODIFICATIONS.md +1 -1
- README.md +1 -1
- decision_config.json +1 -1
MODIFICATIONS.md
CHANGED
|
@@ -7,7 +7,7 @@ Parent: decisionmodels/reax-1.1 @ 142b19355af00251f08db3d287307dd8b711c133
|
|
| 7 |
|---|---|
|
| 8 |
| model weights (`model-*.safetensors`) | LoRA fine-tune(s) merged into the backbone (4 run(s), soup norm mean); final RMSNorm output scaled x2.8 (logit calibration for the fixed SN92 readout temperature) |
|
| 9 |
| readout head (`readout.safetensors`) | fine-tuned letter rows A-Z (letter rows centred: the same vector subtracted from every row, probabilities unchanged) |
|
| 10 |
-
| `decision_config.json` | temperature scaled x2.8
|
| 11 |
| `DATA_SOURCES.md` | added (training data sources, revisions and licences) |
|
| 12 |
| `README.md` | rewritten |
|
| 13 |
| `reax-specialist.json` | added (SN92 manifest) |
|
|
|
|
| 7 |
|---|---|
|
| 8 |
| model weights (`model-*.safetensors`) | LoRA fine-tune(s) merged into the backbone (4 run(s), soup norm mean); final RMSNorm output scaled x2.8 (logit calibration for the fixed SN92 readout temperature) |
|
| 9 |
| readout head (`readout.safetensors`) | fine-tuned letter rows A-Z (letter rows centred: the same vector subtracted from every row, probabilities unchanged) |
|
| 10 |
+
| `decision_config.json` | temperature 10.5667: scaled x2.8 with the logits, then refitted on held-out native-protocol rows (x1.194) so native probabilities are as well calibrated as the base's |
|
| 11 |
| `DATA_SOURCES.md` | added (training data sources, revisions and licences) |
|
| 12 |
| `README.md` | rewritten |
|
| 13 |
| `reax-specialist.json` | added (SN92 manifest) |
|
README.md
CHANGED
|
@@ -24,7 +24,7 @@ Perplexity AI, the Qwen team or REAX.
|
|
| 24 |
Unchanged from the base: joint protocol v3 (template `joint-question-inline-readout-v1`), noncausal full attention,
|
| 25 |
one readout per question via the separate readout head (`readout.safetensors`, [255, 5120]); probabilities are a
|
| 26 |
softmax over the option codes. Loading uses the bundled, unmodified `pplx_decider` package in `source/src/`
|
| 27 |
-
(Perplexity AI, Apache-2.0). The fitted logit scale (x2.8) is folded into the backbone's final RMSNorm (equivalent to scaling every readout row), and the saved temperature in `decision_config.json` is scaled by the same factor
|
| 28 |
|
| 29 |
## Training
|
| 30 |
LoRA fine-tuning (all linear layers of the text backbone) plus the readout letter rows, on decisionmodels/sn92-generalist-train-v1 and decisionmodels/sn92-specialists (both CC-BY-4.0) plus public permissively licensed datasets listed in DATA_SOURCES.md, in the SN92 single-question prompt and the native pplx joint format; weighted soup of several runs; the logit scale fitted on held-out items.
|
|
|
|
| 24 |
Unchanged from the base: joint protocol v3 (template `joint-question-inline-readout-v1`), noncausal full attention,
|
| 25 |
one readout per question via the separate readout head (`readout.safetensors`, [255, 5120]); probabilities are a
|
| 26 |
softmax over the option codes. Loading uses the bundled, unmodified `pplx_decider` package in `source/src/`
|
| 27 |
+
(Perplexity AI, Apache-2.0). The fitted logit scale (x2.8) is folded into the backbone's final RMSNorm (equivalent to scaling every readout row), and the saved temperature in `decision_config.json` is scaled by the same factor and then refitted on held-out native-protocol rows (temperature 10.5667), so native probabilities stay calibrated.
|
| 28 |
|
| 29 |
## Training
|
| 30 |
LoRA fine-tuning (all linear layers of the text backbone) plus the readout letter rows, on decisionmodels/sn92-generalist-train-v1 and decisionmodels/sn92-specialists (both CC-BY-4.0) plus public permissively licensed datasets listed in DATA_SOURCES.md, in the SN92 single-question prompt and the native pplx joint format; weighted soup of several runs; the logit scale fitted on held-out items.
|
decision_config.json
CHANGED
|
@@ -4,7 +4,7 @@
|
|
| 4 |
"revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
|
| 5 |
"attention_mode": "noncausal_full_attention",
|
| 6 |
"pooling": "last",
|
| 7 |
-
"temperature":
|
| 8 |
"max_pixels": 1638400,
|
| 9 |
"codes": [
|
| 10 |
"A",
|
|
|
|
| 4 |
"revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
|
| 5 |
"attention_mode": "noncausal_full_attention",
|
| 6 |
"pooling": "last",
|
| 7 |
+
"temperature": 10.5667,
|
| 8 |
"max_pixels": 1638400,
|
| 9 |
"codes": [
|
| 10 |
"A",
|