AstroHan/decision-head-qwen3.5-4b-rlcd-32k

Decision head for Qwen3.5-4B — RLCD, 32K one-pass

The common read on Jev is that its generalisation comes from the data rather than the training algorithm. This adapter is one arm of an experiment built to test that: base, algorithm and hyperparameters held fixed, three seeds, held-out sets grouped by their relationship to the training data. The answer is in the transfer table below — coverage predicts generalisation in both directions, and where coverage is missing, RL lands below the untrained base.

A LoRA adapter that turns Qwen3.5-4B into a typed decision model: one forward pass, no generation, a probability distribution read off the option-letter logits. 4.9M trainable parameters (0.12% of the base), attention projections only — no MLP, no embeddings, no output head. The classification head is not new weights: it reads single-token A–Z logits out of the frozen vocabulary.

Trained with RLCD (Gaussian noise on candidate logits as the action, leave-one-out baseline over 8 samples) on 32,000 items in a single pass. Three seeds were trained; this is seed 1, selected on a held-out 400-item validation split (0.8675 vs 0.8500 and 0.8650) before any test set was consulted. Seed 3 scores higher on JevBench (0.797 vs 0.779) — picking it would have been selection on the test set, so it is reported below but not released as the default.

What it is for

Bounded decisions where the caller supplies the material, the question and the exact option set, and wants a calibrated-ish distribution over those options: request routing, answer-adequacy judging, policy yes/no checks, intent classification, ordinal severity scoring, evidence support.

It does not generate text. It cannot be prompted into free-form answers. Input is capped at 4,096 tokens and is not truncated — a longer input raises rather than silently losing material.

Results

Against the untrained base on the frozen held-out sets:

Set	n	base	seed 1	seed 2	seed 3
in-distribution	3,433	0.686	0.955	0.939	0.942
out-of-distribution	4,637	0.670	0.927	0.919	0.903
wide option sets (up to 77)	1,200	0.563	0.872	0.864	0.790
public human-labelled panel	3,880	0.717	0.749	0.728	0.721
BFCL V4 (cleaned)	3,206	0.897	0.934	0.938	0.929
RewardBench 2	1,802	0.675	0.721	0.708	0.718

JevBench v1.2 (external, 231 public items)

JevBench is run by a third party; we did not write its items. Scored with its own harness through an adapter that mirrors semif_direct, the Qwen3.5-4B entrant already on its board, so the two are comparable item for item.

	easy 48	standard 72	hard 111	all 231
base (untrained)	0.979	0.861	0.577	0.749
seed 1 (this model)	1.000	0.986	0.550	0.779
seed 3	1.000	0.986	0.586	0.797
SemIf (same base model)	1.000	0.986	0.613	0.810
Jev 1.13.0	1.000	0.986	0.730	0.866

On the easy and standard tiers this ties Jev item for item. The whole remaining gap is the hard tier.

The part worth reading: transfer has a direction

The headline numbers hide a split that three independent evaluations agree on. Grouping held-out tasks by their relationship to the training stream:

relationship to training data	site	base	seed 1 (this model)	seeds 2 / 3	Jev
same dataset, held-out split	massive ×2, paws (mean)	80.8	87.9	88.0 / 86.6	88.0
same structure, similar material	JevBench probability	0.30	0.50	0.50 / 0.60	0.70
same structure, similar material	JevBench trap	0.88	1.00	1.00 / 1.00	1.00
same structure, different material	pubmedqa	70.8	65.2	60.0 / 62.8	—
same structure, different material	vitaminc-dev	73.8	66.3	60.4 / 60.3	—
structure absent: long, many interacting rules	JevBench long_policy	0.47	0.32	0.37 / 0.26	0.63

Six rows, three seeds, same sign in every cell. RL on a narrow slice of a structure does not merely fail to generalise to other material of that structure — it lands below the untrained starting point. The algorithm never varied, so the variable is the data, and narrow data is a liability rather than a no-op.

Two cells that look more dramatic in one seed are deliberately left out of the table: JevBench multi_hop rises 0.50 → 0.83 and tradeoff falls 0.83 → 0.33 in seed 3 only (seed 1: 0.56 and 0.83). An earlier version of this card showed those seed-3 cells next to seed-2 public-panel cells under a single "after RL" column; that was a cross-seed selection and is corrected here.

Calibration also degrades: Brier 0.342 → 0.378 on JevBench, and expected calibration error rises on the public panel. The model becomes both more accurate and more overconfident.

If you plan to use this on material that does not look like ordinary web/wiki prose, or on decisions that require weighing competing objectives, measure against the untrained base before adopting it.

Known limits

• No ablation arm. This run changed data, reward, base model and learning rate at once. Gains are not attributable to any one of them.
• Adapter capacity is a live confound. rank 8, attention-only, inherited unexamined from a 0.6B prototype. Whether the regressions above come from narrow data or from an adapter too small to hold two strategies is not separable from this run alone.
• One-pass stream, one frozen order. All three seeds share the same data order; seeds vary only LoRA init and RLCD noise, so run-to-run variance is understated.
• Arithmetic and dates are out of reach. JevBench temporal_numeric: base 0.27, after RL 0.13, Jev 0.27. One forward pass with no intermediate computation cannot do this, for anyone.
• Small cells. JevBench long_policy is 19 items, tradeoff 6. The regression below base rests on three sites agreeing across three seeds (pubmedqa, vitaminc, long_policy), not on any one cell. The tradeoff drop, which would speak to "no rule decides" items directly, is seed-dependent and is not part of the claim.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE, REV = "Qwen/Qwen3.5-4B", "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"
tok = AutoTokenizer.from_pretrained(BASE, revision=REV)
model = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(model, "AstroHan/decision-head-qwen3.5-4b-rlcd-32k").eval()

The prompt format matters: the adapter was trained on one exact template (system instruction, then Material: / Question: / Options: with single-token letter labels, chat template, no thinking). decision.py in the code repository is the reference implementation — reproduce it exactly or the letter logits will not mean what they meant in training.

Training data

32,000 items in a single pass: 29,000 choice, 1,800 ordinal (3/4/5/6 levels), 1,200 boolean, drawn from eight synthetic families plus public datasets.

The mixture is not redistributed. It derives from public datasets under differing licences and from synthetic items generated through commercial APIs. The frozen manifest (SHA-256, per-family proportions, per-domain concentration, prefix drift) and the build scripts are in the code repository, and the snapshot rebuilds byte-identically from them with your own API keys.

Two ways the mixture is not diverse, measured and recorded before the run: routing has only 3 domains (largest 33.8%) and evidence only 4 (25.1%); the interleaver balances stream buckets but not sources, so a 512-item prefix has source TV distance 0.059 from the final stream. Both turned out to matter — see the transfer table above.

Compute

One H200 SXM, 9.0 GPU-hours total for three seeds plus all evaluation.

Citation

Full report with the five pre-registered predictions and their resolutions: see REPORT.md in the code repository. The criteria were written before the run; two resolved against expectation and one revealed a flaw in its own criterion, all of which the report states.

License

Apache 2.0, matching the base model. JevBench results are reproduced from its MIT-licensed repository; JevBench is not affiliated with TypeSafe AI, whose Jev model it measures.