guychuk commited on
Commit
ed45bdd
·
verified ·
1 Parent(s): dd605dc

Card: document the long-document state-ordering caveat (measured, n=605)

Browse files
Files changed (1) hide show
  1. README.md +8 -0
README.md CHANGED
@@ -161,6 +161,14 @@ jevlogs (Loghub research license, caveated), Mind2Web.
161
  DecisionMix v2 (K = 50 fine-grained topics) — the one clear regression at this size.
162
  - Held-out rubric styles and Noul accuracy give back a few points vs. that same untyped 4B baseline,
163
  in exchange for the calibration and external-floor gains above.
 
 
 
 
 
 
 
 
164
  - Inference code is included in this repo; training code release to follow.
165
 
166
  ## License
 
161
  DecisionMix v2 (K = 50 fine-grained topics) — the one clear regression at this size.
162
  - Held-out rubric styles and Noul accuracy give back a few points vs. that same untyped 4B baseline,
163
  in exchange for the calibration and external-floor gains above.
164
+ - **Put the case facts AFTER the policy text on long documents.** This checkpoint was trained on a
165
+ corpus whose long states rendered the case last, and it inherited that ordering. Measured on 605
166
+ held-out long policy states (identical items, only the position of the case block differs, no
167
+ truncation at scoring time): **0.866** with the facts last versus **0.612** with the facts
168
+ first — a **25-point** drop. On the yes/no (Noul) subset the facts-first score is
169
+ 0.555, at the 0.612 majority-class floor, i.e. no better than always answering the
170
+ majority label. Short states are unaffected. A retrained checkpoint without this bias exists
171
+ (`tm2`, +33.8 points facts-first) and will supersede this release.
172
  - Inference code is included in this repo; training code release to follow.
173
 
174
  ## License