Jwen-Soft-27B

A decision model. Give it some state (text or JSON) and a question with a fixed set of options, and it returns a probability for every option instead of generated text. It's a LoRA adapter on Qwen/Qwen3.8-27B, trained with RLCD on soft targets. The aim is probabilities that track how often the answer is actually right, rather than the near-0-or-1 outputs that training on hard labels produces.

Each option gets a one-token code in the prompt. The hidden state at the answer position is scored against those codes' rows of the LM head only, so there's one forward pass per question and no generated text to parse. Up to 255 options are supported.

Try it in the demo Space.

Quick start

hf download aroraaman/Jwen-Soft-27B --local-dir Jwen-Soft-27B
cd Jwen-Soft-27B
pip install -r requirements.txt
python inference.py --requests examples/queries.jsonl

This needs a GPU with about 60 GB of memory; the bf16 base model is 52 GB.

From Python:

from inference import load

model = load(".")
model.ask(
    state={"ticket": "Hi, I was charged twice for my March subscription."},
    questions={
        "queue": {
            "instructions": "Which support queue should handle `ticket`?",
            "criteria": {
                "billing": "Charges, refunds, invoices and payment methods.",
                "technical": "Bugs, errors, outages and product not working.",
                "none": "None of the queues above fits.",
            },
        }
    },
)
# {"queue": {"choice": "billing", "probabilities": {"billing": 0.88, "technical": 0.04, ...}}}

criteria maps each option to a description, or to None; a plain list of options also works. Several questions can share one state. Refer to JSON fields of the state with backticks, as in `ticket`.

jwen_config.json pins the base model revision and the temperature 1.378 fitted on the validation split, and load applies it. Keep it: at temperature 1 the probabilities are overconfident by about 8 points.

Prompts longer than 65,536 tokens are refused, never truncated. For faster long prompts, install flash-linear-attention 0.5.2 and causal-conv1d. On Hopper GPUs these need Triton 3.7.1 or newer.

Training

• Data: 53,379 training rows from 23 public sources:
  • LLM-as-judge preference and rating data: HelpSteer2 and HelpSteer3, JudgeLM, MT-Bench human judgments, AgentJudgeBench.
  • Data with annotator disagreement: ChaosNLI, Measuring Hate Speech, GoEmotions, SST.
  • Multiple-choice QA, intent and function-calling data.
• Soft targets: vote shares smoothed as a Dirichlet posterior mean; rating means split across the two adjacent levels; 0.95 on gold labels.
• Splits and decontamination: splits are group-disjoint, and every row sharing a normalised line of 32 or more characters with the Decision Index suite was dropped.
• Objective: RLCD, a Gaussian policy over the option logits.
  • Each question gets 8 samples, each rewarded with a proper scoring rule: log score plus 0.5 × spherical score, minus the ranked probability score on ordered ratings.
  • The update is a group-normalised policy gradient, with soft cross-entropy as an anchor.
  • The sampling noise decays from 0.5 to 0.1.
• Setup: LoRA rank 16, learning rate 5e-5, effective batch 16, one epoch (3,000 steps) on one H100.
• Checkpoint choice: the final checkpoint was chosen over the step-1,500 one by validation agreement. That rule was fixed before looking at test or benchmark results.

Results

Held-out test split (5,921 questions), at temperature 1.378:

Metric	Value
Soft cross-entropy	0.795
Agreement with the target's top option	78.4%
Expected calibration error	0.030 (0.105 at T = 1)
Brier score	0.228

On a separate frozen holdout of 324 decision questions, it gets 309 right (95.4%).

The table below is an approximate score on the Jev Decision Index, not an official leaderboard entry:

• It uses a fixed 3,600-request sample of the suite, about 81 requests per benchmark.
• Every column is scored with the same recipe over 31 shared benchmarks.

Area	Jwen-Soft-27B	Jev 1.13.0	AutoJev-27B	Qwen3.8-27B base
Knowledge	0.388	0.472	0.401	0.255
Language	0.627	0.626	0.629	0.587
Retrieval	0.470	0.466	0.452	0.421
Tools	0.684	0.653	0.689	0.639
Arts	0.349	0.340	0.365	0.254
Index	50.4	51.1	50.7	43.1

On the complete BFCL benchmark (1,694 cases), it scores 0.980 case-exact. That ties for best on the leaderboard; Jev scores 0.958 and AutoJev-27B 0.976.

Example outputs

These come from examples/queries.jsonl, written for this card rather than taken from any benchmark. All are at the release temperature.

Query	Top choice	Probabilities
Ticket: charged twice, wants a refund	billing	billing 0.88, others 0.04 each
Ticket: can't log in and was charged	billing	billing 0.49, account 0.39, technical 0.10
Judge: a correct and a wrong answer on reversing a list	A	A 0.85, B 0.08, tie 0.07
Judge: two reasonable coffee-shop names	B	B 0.45, tie 0.37, A 0.19
Tool for "weather in Lisbon tomorrow"	get_forecast	0.87
Restaurant intent for "recommend a book on Rome"	none	0.89
Rate "seasons are caused by distance to the Sun" (1–5)	1	1 0.68, 2 0.11, 3 0.09
Does max starting at 0 handle all lists?	no	0.91
Checkout returning 500 errors for all customers	critical	0.86
Refund eligible? Unopened, purchase date not given	yes	yes 0.87, no 0.13
Will it rain in London on a random April day?	yes	yes 0.92, no 0.08

It hedges well when the input itself is ambiguous. Three things to know:

• Confident answers top out near 0.9, with about 0.04 on each other option. That comes from the training targets, where gold labels were smoothed to 0.95, and from the fitted temperature. Read 0.85 or more as "confident", not 0.99.
• It doesn't reliably notice a missing fact. With the purchase date absent, the refund question should be close to 50/50, but it answers "yes" at 0.87. That's barely below the case where the date is given. When a decision depends on a specific fact, ask a separate question such as "does the state say when the item was bought?"
• Real-world odds aren't calibrated. Its probabilities reflect how clear the answer is from the input, not how often an event happens in the world. It says 0.92 for rain on a random April day in London, which is really close to a coin flip.

Limitations

• "None of these" is over-predicted on some intent tasks. On SGD it picks the no-match option for most utterances, scoring 0.22 against 0.43 for Jev. When a no-match option is offered, check its probability instead of trusting the top choice.
• Knowledge and reasoning trail Jev: GPQA Diamond is 0.51 against Jev's 0.78, and MuSR and WinoGrande are also lower. The model scores options and doesn't reason step by step.
• Choice questions only. Yes/no and rating questions are asked as choices.
• Calibration was measured on held-out data from the training distribution. Re-check it on a new domain before using the probabilities as thresholds.

License

The adapter weights are released under CC BY-NC 4.0, because two training sources are non-commercial: JudgeLM-100K text and ChaosNLI (both CC BY-NC 4.0). The base model and the code files inference.py and schema.py are Apache 2.0; see LICENSE-APACHE.