pngwn/system-one-qwen3.5-4b-scorer-v2b

System One scorer: Qwen/Qwen3.5-4B-Base + scalar scoring head

Scores each option of a typed question (yes/no, choice, score) in a single forward pass and softmaxes per question. No autoregressive generation; the output space is exactly the option set supplied by the caller.

Training code: system_one.py in this repository.

Training

• data: pngwn/system-one-decisions (train questions 45932, epochs 2)
• LoRA r=16, seq len 384, option cap 16, lr 0.0001
• trainable params 30476800 of 4236230656; 11484 steps in 13537s
• temperature 2.350 (fitted on val)

Val split

task	n	accuracy	ECE	Brier
choice	1176	0.634	0.073	0.489
escalate	341	0.915	0.066	0.136
noul	227	0.943	0.025	0.076
review	342	0.743	0.052	0.339
severity	345	0.751	0.126	0.360
team	348	1.000	0.000	0.000
workflow4	2308	0.845	0.048	0.201
ALL	5087	0.803	0.022	0.264

Limitations

• Trained with option sets capped at 16 for the high-cardinality tasks (banking77 77 options, ticket routing 52); evaluation scores all options.
• Sequence length 384 truncates long states (MMLU questions and long reviews).
• The ticket component of the training data is CC-BY-NC-4.0, so this model inherits a non-commercial restriction.
• Accuracy is far below a frontier model; the claim here is calibrated distributions over a fixed option set at single-pass latency.