ebobo commited on
Commit
42dc016
·
verified ·
1 Parent(s): 00855bd

native temperature refit (10.5667)

Browse files
Files changed (3) hide show
  1. MODIFICATIONS.md +1 -1
  2. README.md +1 -1
  3. decision_config.json +1 -1
MODIFICATIONS.md CHANGED
@@ -7,7 +7,7 @@ Parent: decisionmodels/reax-1.1 @ 142b19355af00251f08db3d287307dd8b711c133
7
  |---|---|
8
  | model weights (`model-*.safetensors`) | LoRA fine-tune(s) merged into the backbone (4 run(s), soup norm mean); final RMSNorm output scaled x2.8 (logit calibration for the fixed SN92 readout temperature) |
9
  | readout head (`readout.safetensors`) | fine-tuned letter rows A-Z (letter rows centred: the same vector subtracted from every row, probabilities unchanged) |
10
- | `decision_config.json` | temperature scaled x2.8, the same factor as the logits (native probabilities unchanged) |
11
  | `DATA_SOURCES.md` | added (training data sources, revisions and licences) |
12
  | `README.md` | rewritten |
13
  | `reax-specialist.json` | added (SN92 manifest) |
 
7
  |---|---|
8
  | model weights (`model-*.safetensors`) | LoRA fine-tune(s) merged into the backbone (4 run(s), soup norm mean); final RMSNorm output scaled x2.8 (logit calibration for the fixed SN92 readout temperature) |
9
  | readout head (`readout.safetensors`) | fine-tuned letter rows A-Z (letter rows centred: the same vector subtracted from every row, probabilities unchanged) |
10
+ | `decision_config.json` | temperature 10.5667: scaled x2.8 with the logits, then refitted on held-out native-protocol rows (x1.194) so native probabilities are as well calibrated as the base's |
11
  | `DATA_SOURCES.md` | added (training data sources, revisions and licences) |
12
  | `README.md` | rewritten |
13
  | `reax-specialist.json` | added (SN92 manifest) |
README.md CHANGED
@@ -24,7 +24,7 @@ Perplexity AI, the Qwen team or REAX.
24
  Unchanged from the base: joint protocol v3 (template `joint-question-inline-readout-v1`), noncausal full attention,
25
  one readout per question via the separate readout head (`readout.safetensors`, [255, 5120]); probabilities are a
26
  softmax over the option codes. Loading uses the bundled, unmodified `pplx_decider` package in `source/src/`
27
- (Perplexity AI, Apache-2.0). The fitted logit scale (x2.8) is folded into the backbone's final RMSNorm (equivalent to scaling every readout row), and the saved temperature in `decision_config.json` is scaled by the same factor, so native probabilities equal those of the fine-tuned model at its fitted temperature.
28
 
29
  ## Training
30
  LoRA fine-tuning (all linear layers of the text backbone) plus the readout letter rows, on decisionmodels/sn92-generalist-train-v1 and decisionmodels/sn92-specialists (both CC-BY-4.0) plus public permissively licensed datasets listed in DATA_SOURCES.md, in the SN92 single-question prompt and the native pplx joint format; weighted soup of several runs; the logit scale fitted on held-out items.
 
24
  Unchanged from the base: joint protocol v3 (template `joint-question-inline-readout-v1`), noncausal full attention,
25
  one readout per question via the separate readout head (`readout.safetensors`, [255, 5120]); probabilities are a
26
  softmax over the option codes. Loading uses the bundled, unmodified `pplx_decider` package in `source/src/`
27
+ (Perplexity AI, Apache-2.0). The fitted logit scale (x2.8) is folded into the backbone's final RMSNorm (equivalent to scaling every readout row), and the saved temperature in `decision_config.json` is scaled by the same factor and then refitted on held-out native-protocol rows (temperature 10.5667), so native probabilities stay calibrated.
28
 
29
  ## Training
30
  LoRA fine-tuning (all linear layers of the text backbone) plus the readout letter rows, on decisionmodels/sn92-generalist-train-v1 and decisionmodels/sn92-specialists (both CC-BY-4.0) plus public permissively licensed datasets listed in DATA_SOURCES.md, in the SN92 single-question prompt and the native pplx joint format; weighted soup of several runs; the logit scale fitted on held-out items.
decision_config.json CHANGED
@@ -4,7 +4,7 @@
4
  "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
5
  "attention_mode": "noncausal_full_attention",
6
  "pooling": "last",
7
- "temperature": 8.8497,
8
  "max_pixels": 1638400,
9
  "codes": [
10
  "A",
 
4
  "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
5
  "attention_mode": "noncausal_full_attention",
6
  "pooling": "last",
7
+ "temperature": 10.5667,
8
  "max_pixels": 1638400,
9
  "codes": [
10
  "A",