faizaltkl/qdec-lay-decision-model

Decision model (Qwen3.5-4B + decision head)

A small decision model: instead of generating text, it reads an input and answers structured questions about it with calibrated probabilities. It was fine-tuned with Unsloth's decision-model training, which adds a small scoring head (the same design as Cloudflare's Clef) on top of the LLM.

You define the questions at request time, so the same model can route tickets, flag fraud, detect refund requests or rate urgency without retraining.

Question type	Use it for	Returns
choice	pick one option (e.g. which team)	the chosen option + a probability per option
noul	yes / no	probability of "yes"
score	a level on a scale (e.g. urgency)	the level + a probability per level

Results at a glance (3,006 unseen eval rows, 4,923 decisions): 84.8% of decisions correct, versus 33.2% before fine-tuning.

Quick start

from unsloth import FastDecisionModel

model, tokenizer = FastDecisionModel.from_pretrained("faizaltkl/decision-model", load_in_4bit=True)
FastDecisionModel.for_inference(model)

answers = FastDecisionModel.predict(
    model, tokenizer,
    "I was charged twice for my subscription, please refund one of them today.",
    {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this?",
            "criteria": {
                "billing": "invoices, payments, refunds",
                "technical": "bugs, outages, errors",
                "sales": "pricing, new plans",
            },
        },
        "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
        "urgency": {"type": "score", "instructions": "How urgent is this?",
                    "criteria": ["not urgent", "soon", "today"]},
    },
)
for name, a in answers.items():
    print(name, a["answer"], a["probabilities"])

The input (state) can be plain text or any JSON object. Act on the probabilities (for example, only auto-route when the top option is above 0.8) rather than on the single answer.

Evaluation

All results are on held-out eval rows that were never used for training or calibration: 3,006 rows (4,923 individual decisions) from each source's official test split, plus held-out hand-written rows. The base model and the fine-tuned model were scored on exactly the same rows.

The base model uses the same decision head before training (randomly initialised), so its scores show the starting point, not the ability of Qwen itself.

Overall

	Base	Fine-tuned	Change
All decisions (4,923)	33.2%	84.8% (±1.0%)	+51.6 pts
Rows with every answer correct (3,006)	29.0%	80.8%	+51.8 pts
Choice questions (2,690)	29.6%	86.5% (±1.3%)	+57.0 pts
Yes/no questions (1,143)	53.5%	90.1% (±1.7%)	+36.7 pts
Score questions (1,090)	21.0%	75.0% (±2.6%)	+53.9 pts

± is the 95% margin of error.

Per eval set

Eval set	Decisions	Base	Fine-tuned	± (95%)	Change	Rows fully correct	Calibration error (ECE)
Business decisions
typed_decisions	2000	30.1%	79.6%	1.8%	+49.5	39.0%	0.164
synth_customer_support	120	60.0%	93.3%	4.5%	+33.3	70.8%	0.042
synth_banking	115	34.8%	91.3%	5.1%	+56.5	56.5%	0.040
synth_ecommerce	88	27.3%	83.0%	7.9%	+55.7	54.5%	0.057
synth_it_hr	84	33.3%	88.1%	6.9%	+54.8	57.1%	0.059
Intent & topic
banking77 (12 options)	200	22.0%	96.0%	2.7%	+74.0	96.0%	0.024
clinc150 (12 options)	200	13.0%	96.5%	2.5%	+83.5	96.5%	0.032
ag_news	200	45.5%	94.0%	3.3%	+48.5	94.0%	0.063
Reading & reasoning
boolq	200	58.5%	95.0%	3.0%	+36.5	95.0%	0.033
mnli	200	37.5%	89.5%	4.2%	+52.0	89.5%	0.066
snli	200	50.0%	94.0%	3.3%	+44.0	94.0%	0.066
wanli	200	36.0%	73.0%	6.2%	+37.0	73.0%	0.061
arc (challenge)	200	32.0%	96.5%	2.5%	+64.5	96.5%	0.044
arc_easy	200	33.0%	99.0%	1.4%	+66.0	99.0%	0.014
commonsense_qa	200	25.0%	85.0%	4.9%	+60.0	85.0%	0.056
mmlu	200	31.0%	73.5%	6.1%	+42.5	73.5%	0.118
Sentiment & safety
sst5	200	23.0%	59.5%	6.8%	+36.5	59.5%	0.156
prompt_injections	116	47.4%	100.0%	–	+52.6	100.0%	0.014

Changes are in percentage points. "Rows fully correct" differs from accuracy only where a row asks several questions (typed_decisions and the hand-written sets ask 4–5). ECE (expected calibration error) is how far the model's confidence is from how often it is actually right; lower is better, and under about 0.05 is well calibrated.

Reading the results

• typed_decisions improved from 73.85% to 79.6% compared with an earlier 100-step training run, on the same 2,000-decision test split. For context, Unsloth reports 73% (Qwen3.5-0.8B) and 78% (Qwen3.5-2B) on typed-decisions after training; the setups differ, so this is a rough reference only.
• Strongest: intent routing, yes/no reading questions, science questions and prompt-injection detection (94–100%).
• Weakest: SST-5 sentiment (59.5%; five subjective levels), WANLI (73%; deliberately hard reasoning pairs) and MMLU (73.5%; knowledge-heavy).
• Confidence is less reliable on typed_decisions, SST-5 and MMLU (ECE 0.12–0.16). typed_decisions has soft labels where annotators disagreed, which makes its probabilities harder to match. Elsewhere ECE is 0.066 or lower.

Training data

Trained on faizaltkl/decision-model-dataset: 37,799 training rows in the decision format (state / questions / gold), mixing:

Group	Sources	Question types
Business workflows	LocalLLaMA/typed-decisions (customer service, invoices, security incidents, agent traces; soft multi-annotator labels)	choice, yes/no, score
Hand-written domain data	451 examples for customer support, banking & payments, e-commerce and IT/HR, 4–5 questions each, written with an AI assistant	choice, yes/no, score
Intent classification	BANKING77, CLINC150	choice
Topic & sentiment	AG News, SST-5	choice, score
Reading & reasoning	BoolQ, MNLI, SNLI, WANLI, ARC, CommonsenseQA, MMLU	choice, yes/no
Safety	deepset prompt-injections	yes/no

Up to 3,000 rows per public source. typed-decisions and the hand-written data are repeated 3× because they are small. Eval rows come from each source's official test split and were checked to not overlap with training data.

Training procedure

Setting	Value
Base model	unsloth/Qwen3.5-4B, loaded in 4-bit
Method	LoRA (r=16, alpha=16, dropout 0) + decision head, via FastDecisionModel / DecisionTrainer
Trainable parameters	64.5M (2.10% of the parameters reported for the 4-bit model)
Epochs / steps	2 epochs, 2,364 steps
Batch size	8 per device × 4 gradient accumulation = 32
Learning rate	2e-4, cosine schedule, 10 warmup steps, weight decay 0.01
Max sequence length	2,048 tokens
Calibration	FastDecisionModel.calibrate on a held-out validation split
Hardware	1× NVIDIA RTX 4000 (RunPod), about 2.5 hours of training

Intended use

• Routing and triaging messages: support tickets, banking requests, orders, internal IT/HR requests
• Flagging items for review: possible fraud, refund requests, security incidents, urgent cases
• Fast, cheap classification where you want probabilities you can threshold

Limitations

• Not for final high-stakes decisions. Use it to prioritise and route, with a human reviewing anything involving fraud, money, safety, health, employment or legal matters.
• Mostly English. Only a handful of training examples are in other languages.
• BANKING77 and CLINC150 were trained and evaluated with 12 options per question (the correct intent plus 11 random others) to fit the context length. Their 96% scores are therefore not comparable to the standard 77-way and 151-way benchmarks.
• MMLU rows were trained on its test split (it has no training split) and evaluated on its validation split, so MMLU scores are not a clean benchmark.
• The hand-written domain data is small and written in one style (84–120 eval decisions per domain, hence the wider margins), so in-domain scores may be higher than on real customer messages. Test on your own data before relying on it.
• Multi-question rows are harder than single decisions: on typed_decisions only 39% of rows have all five answers right, even though 79.6% of individual answers are.
• Score questions are the least accurate (75%), as the levels are often subjective.
• Many options in one choice question share a fixed token budget; above about 20 described options, labels may be trimmed. Keep option lists short.

License

The fine-tuned weights are subject to the license of the base model (Qwen3.5-4B). The training data mixes datasets with their own licenses (some share-alike or non-commercial); check each source before commercial use.

Acknowledgements

Built with Unsloth. Training data includes LocalLLaMA/typed-decisions, BANKING77, CLINC150, AG News, SST-5, BoolQ, MNLI, SNLI, WANLI, ARC, CommonsenseQA, MMLU and deepset/prompt-injections; thanks to their authors.