How v1 was built and checked
Training data
v1 uses pool v6 lite: 50,464 records (56,280 questions). 26,991 records are a seeded sample of an earlier pool. 23,473 are new: the training splits of 65 families from the Bekko System One dataset (revision 9c673a09, capped per family) plus three extra label sets with 50 to 100 options (trec_fine, ledgar, dbpedia level 2). 10.3% of questions have 21 to 160 options, so v1 is trained on wide questions, and its prompt format supports up to 255 options.
Before training, new records whose input matched an S1MB English test input were removed: 113 exact matches, 2,885 by 8-word shingle overlap (Jaccard at least 0.5) and 1,526 where at least half of a test input's shingles sat inside the training item (4,524 in total). Only inputs were compared; test labels were never read. An audit of the final pool still finds 304 of 15,609 test cases with an approximate match, mostly templated generator families (snake 77, spartqa 78, ir-control 32, typed-decisions 24, browser-control 23, silent-failure 23, sponsor-segment 22). We list these as approximate-match candidates and do not claim they are confirmed uses.
Sources whose labels came from an LLM (gretel_pii, laya typed-decisions, LLM-jury aegis2 labels) were left out of the new data. The sampled part still contains typed-decisions and WANLI records (see data and licences).
Release check (set before any v1 score existed)
A v1 model is released only if its accuracy on held-out workflow groups is at least that of the previous release at the same size. The groups come from the same families v1 trained on (other groups), so the check favours v1 and only guards against a broken run. It does not measure how well a model handles a new workflow.
| Size | Held-out accuracy | Result |
|---|---|---|
| 14B | 82.8 | released |
| 4B | 69.9 | released (no clear difference from the previous release; log-loss is worse) |
| 1.7B | 59.3 | not released |
Readout experiments that did not help
We asked whether a cheap change to how the answer is read out could lift the S1MB-style scores without retraining: Noul by balanced accuracy, Choice by probability mass on the top answer, Score by expected-value error. 79 candidates were tried on open development data scored by five model sizes: threshold offsets (fixed and fitted), position and answer-margin rules for Choice, median, mode, quantile and shrinkage readouts for Score, LoRA scale 0.8 and 1.2, eight option orders instead of four, and an empty-input calibration. The pass rule was fixed in advance (mean gain at least 0.005, better on at least 4 of 5 sizes, no loss at 14B). None passed, so v1 keeps the plain readout.
Why: Score gold labels are mostly soft distributions and the metric compares expected values, so the expected value is already the right point estimate; Noul bias runs in opposite directions in different workflows, so one offset helps some and hurts others; and what remains for Choice is a per-workflow preference for certain answers that a single question does not reveal. Calibration temperatures improve probability quality but cannot change any picked answer, so they do not move S1MB scores. What would help is information about the workflow itself (a few labelled examples per workflow) or more diverse training workflows.