Card: document the long-document state-ordering caveat (measured, n=605)
Browse files
README.md
CHANGED
|
@@ -161,6 +161,14 @@ jevlogs (Loghub research license, caveated), Mind2Web.
|
|
| 161 |
DecisionMix v2 (K = 50 fine-grained topics) — the one clear regression at this size.
|
| 162 |
- Held-out rubric styles and Noul accuracy give back a few points vs. that same untyped 4B baseline,
|
| 163 |
in exchange for the calibration and external-floor gains above.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
- Inference code is included in this repo; training code release to follow.
|
| 165 |
|
| 166 |
## License
|
|
|
|
| 161 |
DecisionMix v2 (K = 50 fine-grained topics) — the one clear regression at this size.
|
| 162 |
- Held-out rubric styles and Noul accuracy give back a few points vs. that same untyped 4B baseline,
|
| 163 |
in exchange for the calibration and external-floor gains above.
|
| 164 |
+
- **Put the case facts AFTER the policy text on long documents.** This checkpoint was trained on a
|
| 165 |
+
corpus whose long states rendered the case last, and it inherited that ordering. Measured on 605
|
| 166 |
+
held-out long policy states (identical items, only the position of the case block differs, no
|
| 167 |
+
truncation at scoring time): **0.866** with the facts last versus **0.612** with the facts
|
| 168 |
+
first — a **25-point** drop. On the yes/no (Noul) subset the facts-first score is
|
| 169 |
+
0.555, at the 0.612 majority-class floor, i.e. no better than always answering the
|
| 170 |
+
majority label. Short states are unaffected. A retrained checkpoint without this bias exists
|
| 171 |
+
(`tm2`, +33.8 points facts-first) and will supersede this release.
|
| 172 |
- Inference code is included in this repo; training code release to follow.
|
| 173 |
|
| 174 |
## License
|