Qwen-9B-decision

An open-weight decision model built with Summit: a full fine-tuned Qwen3.5-9B causal parent and a Typed Decisions QLoRA specialist adapter, together in one repository.

Given a state, a question and candidate options, the model scores the options and returns a probability distribution and selected option. This is constrained decision scoring, not an arbitrary JSON-generation service or a replica of Jev.

Use the parent plus qlora/ for the specialist result. Loading only the repository-root model loads the parent, which scores 61.1%, not 75.8%.

Benchmark: Typed Decisions

The specialist achieves 75.8% accuracy (1,516 / 2,000 decisions) on the held-out Typed Decisions test set: 3.1 percentage points above the published Jev 1.13.0 reference of 72.7%.

Model	Training exposure	Request shape	Accuracy
Qwen-9B-decision parent	No Typed Decisions training	One question per prompt	61.1%
Jev 1.13.0, published reference	General, zero-shot	Five questions per request	72.7%
Qwen-9B-decision + QLoRA	Trained on Typed Decisions train split	One question per prompt	75.8%

This is a reference comparison, not a matched head-to-head win. The specialist has seen this benchmark's training distribution; Jev's published row is zero-shot. The official protocol sends all five questions in one request, whereas our evaluation scores each question separately. The benchmark authors explicitly warn that request shape affects results. We did not rerun Jev, and this is not an official leaderboard submission or evidence of general superiority to Jev. The pinned benchmark card is the source for Jev's result and protocol.

The test has 400 cases across customer service, invoice processing, security incidents and agent-trace observability, with five decisions per case. Gold distributions average three benchmark-teacher samples: accuracy measures agreement with those labels, not objective correctness. No test cases were used for specialist training or validation. The case-bootstrap 95% interval for specialist accuracy is 73.65–77.90% (10,000 resamples, seed 42); it is not a paired significance test against Jev.

Additional specialist metrics: soft-target Brier 0.05413, KL from gold 0.09613, raw ECE 0.13244, and ordinal-score MAE 0.22709. See full metrics, parent metrics and the per-decision outputs in eval/. These are the saved October 1, 2026 evaluations; publication did not retrain or rerun the model.

What is the base model?

The upstream model is Qwen/Qwen3.5-9B, pinned at c202236235762e1c871ad0ccb60c8ee5ba337b9a.

This release contains its text-only causal language-model component, trained through Summit, with 8,953,803,264 parameters. The configuration is Qwen3_5ForCausalLM / qwen3_5_text; no vision encoder is included. The parent is a full fine-tuned checkpoint, not merely the original upstream weights.

The parent weights are preserved in their original FP32 safetensors format (35.82 GB). For inference, explicitly load them in BF16 as shown below. The QLoRA adapter adds 58,195,968 trainable parameters and is approximately 223 MiB. It depends on this exact fine-tuned parent, not directly on the untuned upstream Qwen checkpoint.

How it was trained with Summit

Summit provides the YAML recipes, dataset validation, optional model adapters, training/evaluation loop, cloud orchestration and checkpoint save/reload checks. These two stages use its offline decision recipe, not token-level on-policy distillation, NexRL or SGLang.

Stage 1: full fine-tuning of the causal parent

• Data: 512 converted CLINC150/ContractNLI training decisions. One additional option-order permutation per decision produced 1,024 training views; 256 independent development decisions were used for comparison and selection.
• Teacher: native-reasoning answers from Qwen Max, requested through the Fireworks alias accounts/fireworks/models/qwen3p8-max. Completed answers were encoded as one-hot hard labels. These are not teacher logits or calibrated confidence distributions; the teacher deployment itself was not weight-pinned.
• Objective: 0.5 × reference-label CE + 0.5 × forward KL to the one-hot teacher targets, at temperature 1. Independent reference labels were retained.
• Matched control: reference-only CE and mixed CE/KD started from the same pinned Qwen model. Each arm ran 128 optimizer updates. KD was selected by development accuracy, then NLL, with CE preferred on an exact tie.
• Optimization: full FP32 parameters and AdamW states, BF16 compute, gradient checkpointing, batch 1, accumulation 8, LR 2e-7, gradient norm cap 1, weight decay 0 and seed 42.
• Hardware: one RunPod B300 288 GB, orchestrated by Summit through dstack. The selected KD arm reached 226/256 development decisions (88.28%), versus 223/256 for CE. These development numbers are not held-out test performance.

See the causal training recipe and data pipeline. The public Summit commit documents the workflow; it is not claimed to be the exact historical training checkout. Original source hashes are retained in release provenance.

Stage 2: Typed Decisions specialization with QLoRA

• Starting point: the retained causal KD parent published at this repository's root.
• Data: pinned Typed Decisions revision 561333a8576d22875380b14d25a13065b046538c. The public train split was divided by whole case, stratified by workflow, into 5,400 training decisions / 1,080 cases and 600 validation decisions / 120 cases. All five questions from a case stayed together. Split seed: 20261001.
• Targets: the dataset's existing soft gold distributions, averaged from three benchmark-teacher samples. No new Qwen Max calls were made for this stage.
• Adapter: frozen NF4 double-quantized parent, rank 32, alpha 64, dropout 0.05; LoRA on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
• Objective: forward KL to the benchmark distributions, temperature 1, alpha: 1.0 (no reference-label CE contribution).
• Optimization: one epoch, 675 updates, batch 1, accumulation 8, LR 1e-4, weight decay 0, gradient norm cap 1, seed 42, BF16 compute and FP32 LoRA parameters.
• Hardware: local RTX 5090 32 GB. Training, evaluation, save and reload took about 30 minutes 37 seconds, with 12.64 GiB peak allocated training memory. Both stages completed checkpoint reload verification.

See the specialization recipe and step-by-step guide. The recipes require prepared data and explicit destination/budget settings; the historical teacher cache is not bundled. Fresh teacher acquisition need not yield byte-identical retraining.

Quick start: use the specialist

Install Python 3.12, a suitable NVIDIA driver, and the tested inference dependencies:

pip install torch==2.7.1 --index-url https://download.pytorch.org/whl/cu128
pip install transformers==5.17.0 peft==0.21.1 accelerate==1.15.0 safetensors==0.8.0 huggingface_hub==1.33.0

hf download k4rth33k/Qwen-9B-decision --local-dir checkpoints/Qwen-9B-decision
python checkpoints/Qwen-9B-decision/decision.py \
  --checkpoint checkpoints/Qwen-9B-decision \
  --input checkpoints/Qwen-9B-decision/example_request.json --device cuda

The helper applies the qlora/ adapter by default. Add --base-only for the parent. It uses the same system prompt, non-thinking chat template and restricted answer-letter scoring as the evaluation. It returns option IDs and probabilities; your application can map IDs to validated JSON structures. Inspect the included example input and helper before executing downloaded code.

For reproducible deployment, add --revision COMMIT_SHA to hf download, using an immutable commit from this repository, and keep the entire snapshot together. No access to the original private parent repository is required.

To integrate directly with Transformers/PEFT after downloading the snapshot:

from pathlib import Path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

snapshot = Path("checkpoints/Qwen-9B-decision")
tokenizer = AutoTokenizer.from_pretrained(snapshot, trust_remote_code=False)
base = AutoModelForCausalLM.from_pretrained(
    snapshot, dtype=torch.bfloat16, device_map={"": "cuda:0"},
    attn_implementation="eager", trust_remote_code=False,
)
model = PeftModel.from_pretrained(base, snapshot / "qlora", is_trainable=False).eval()
# Use decision.py's score_decision() for benchmark-compatible option scoring.

The reported test used BF16 parent + unmerged adapter, not an NF4 inference base. Changing quantization, merging precision, prompt, or serving engine can change outputs; re-evaluate before claiming the published score. A 32 GB GPU was used for inference. The original batch-four test peaked at 17.56 GiB allocated memory, not a guarantee for every runtime or input length.

Contents and verification

config.json, model.safetensors          Full causal KD parent (unmerged)
tokenizer.json, tokenizer_config.json   Matching tokenizer
chat_template.jinja                    Matching non-thinking prompt support
qlora/                                 Specialist PEFT config and adapter weights
decision.py, example_request.json       Target-free inference helper and input
eval/                                  Original metrics and per-decision outputs
assets/typed-decisions-accuracy.png     Benchmark reference comparison
release_manifest.json, SHA256SUMS       Provenance, packaging changes and hashes
LICENSE, NOTICE.md                     License and source acknowledgements

The weights are byte-identical to the retained evaluated artifacts:

• Parent SHA-256: 4b29255239d1f1efd8f87d95f790510a9c55e150ca82818fc05f73b30ff102c3
• Adapter SHA-256: 23656838586b2176a1b9f9cea7b6e8059c2c56b9fb505d7338fb00d015366be0

Packaging flattens the parent's model/tokenizer folders and replaces the adapter's machine-local base path with this public repository ID. It does not modify any weight tensors. sha256sum -c SHA256SUMS checks all packaged files except the checksum file itself. The historical evaluator hash and the included helper's hash are recorded separately: the helper is a publication-time implementation, not an exact copy of the historical evaluator.

Limitations, licensing and acknowledgements

This is an English, benchmark-specialized decision model. Its performance on unseen workflows, adversarial state text, general conversation, images or long contexts is not established. Probabilities are not calibrated guarantees; do not use its output as an unchecked high-stakes decision. The specialist was trained with maximum length 768; the test prompts were 148–601 tokens.

Model weights are released under Apache-2.0, with the upstream Qwen license included. See NOTICE.md for Qwen, Summit and dataset attribution, including the separate CLINC150 and ContractNLI dataset licenses. No raw training documents or paid teacher-response text is redistributed here.

Built with Summit, Qwen, PyTorch, Transformers, PEFT, bitsandbytes and dstack. Thanks to the Typed Decisions, CLINC150 and ContractNLI dataset authors.