nikhil061307/jwen_decision

Jwen Decision: a calibrated decision model on Qwen3.5-4B

Jwen Decision reads a piece of text (a support ticket, a conversation, a record) and answers a structured question about it, such as which team should handle this?, is the customer asking for a refund? or how urgent is it?

Instead of generating free text, it returns an answer plus a calibrated probability for every option. You can route automatically when the model is confident, and send the uncertain cases to a person.

Base model	unsloth/Qwen3.5-4B (4-bit, LoRA fine-tuned, merged to 16-bit)
Architecture	Unsloth ClefDecisionModel: LM backbone with a trained decision head
Fine-tuning	LoRA (r=16, α=16) + decision head. 64.5M trainable parameters (2.10%)
Max input length	1,024 tokens
Question types	choice (pick one option), noul (yes/no), score (pick a level)
Trained on	Synthetic customer-support routing data (single-choice questions)
Repo	nikhil061307/jwen_decision

---

Why use it

• Probabilities, not just labels. Every option gets a calibrated score. Confidence is meant to track accuracy.
• Small and cheap to run. About 4B parameters, loadable in 4-bit on a single consumer or free-tier GPU (trained on a Tesla T4).
• Define questions at inference time. You supply the question and the options (criteria) in the call. No retraining is needed to change the label set.
• Several decisions in one call. Ask for team, refund and urgency together.

---

Quick start

from unsloth import FastDecisionModel

model, tokenizer = FastDecisionModel.from_pretrained(
    model_name = "nikhil061307/jwen_decision",
    max_seq_length = 1024,
    load_in_4bit = True,
)
FastDecisionModel.for_inference(model)

answers = FastDecisionModel.predict(
    model,
    tokenizer,
    "Hi, I was charged twice for invoice #4411. Please refund the duplicate today.",
    {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this?",
            "criteria": {
                "billing": "invoices, payments, refunds",
                "technical": "bugs, outages, errors",
                "sales": "pricing, new plans",
            },
        },
        "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
        "urgency": {
            "type": "score",
            "instructions": "How urgent is this?",
            "criteria": ["not urgent", "soon", "today"],
        },
    },
)

for name, result in answers.items():
    print(name, result["answer"], {k: round(v, 3) for k, v in result["probabilities"].items()})

Output from the training notebook:

team    billing  {'billing': 0.998, 'technical': 0.001, 'sales': 0.001}
refund  True     {'true': 0.998, 'false': 0.002}
urgency 2        {'0': 0.001, '1': 0.008, '2': 0.992}

answer is the option key for choice, True/False for noul, and the level index for score.

Requires unsloth, transformers==5.2.0, trl==0.22.2 and torch==2.8.0 (the versions used for training).

---

Performance

Headline results

Evaluation	Examples	Accuracy	ECE ↓	Loss
Before fine-tuning (untrained head, held-out)	400	30.5%	0.104	1.004
After fine-tuning + calibration (held-out, same data source)	400	93.75%	0.044	0.198
External test: LocalLLaMA/typed-decisions (all types)	400	65.55%	0.132	1.289

ECE = expected calibration error: the gap between stated confidence and actual accuracy. Lower is better.

How to read these numbers

• 93.75% is an in-distribution result. The 400 held-out examples come from the same synthetic customer-support data the model trained on, so they show how well it learned that task. They are not a general benchmark.
• 65.55% is the more realistic number for unfamiliar data. On the external typed-decisions test set, which includes question types and phrasing the model did not train on, accuracy and calibration are noticeably worse. record_accuracy (all questions in a record correct) was 13.25%.
• Calibration was fitted on the same 400 held-out rows used to report 93.75% / ECE 0.044. Those two figures are therefore somewhat optimistic. A fresh split would give a fairer estimate.
• 30 of the 400 external test inputs exceeded the 1,024-token limit and were truncated.

Training curve

Epoch	Training loss	Validation loss
1 (300 steps)	0.114	0.192

---

Training details

Data

Source dataset	ZefanCai/Open-Jev (train split, 79,116 rows). Same schema as LocalLLaMA/typed-decisions
Content	Synthetic multi-turn customer-support conversations, e.g. customer-control-v1 (generated conversations are CC0-1.0)
Task	Pick the correct category of a ticket, with explicit rules about priority and ambiguity (account, bug_report, billing, feature_request, …)
Used for training	Random sample of 10,000 rows (seed 3407), split into 9,600 train / 400 held-out
Label construction	Each row's options became the criteria. The label is the option with the highest value in target. Every row was converted into a single choice question named category
Input length	Median 307 tokens, P95 655, max 745. No truncation and no skipped rows

Other splits in the source dataset (not used for training): calibration 4,672 · validation 3,723 · test 10,356 · OOD 15,701.

Hyperparameters

Epochs / steps	1 epoch / 300 steps
Batch size	8 per device × 4 gradient accumulation = 32 effective
Learning rate	2e-4, cosine schedule, 10 warm-up steps
Weight decay	0.01
LoRA	r=16, α=16, dropout 0
Precision	4-bit base weights; float32 activations (Qwen3.5 overflows under fp16 autocast)
Seed	3407
Hardware	1× NVIDIA Tesla T4 (14.5 GB)
Training time	≈ 3 h 11 min
Framework	Unsloth 2026.10.7, Transformers 5.2.0, PyTorch 2.8.0

---

Intended use

Good fits

• Routing and triage: support tickets, emails, requests
• Yes/no flags and urgency or severity levels on short English text
• Human-in-the-loop pipelines that use the probability to decide when to escalate

Not recommended

• Fully automated decisions about people (credit, hiring, insurance, medical, legal) with no human review
• Languages other than English
• Inputs longer than ~1,000 tokens (the end is cut off)
• Safety-critical use

---

Limitations

1. Narrow training distribution. Training used synthetic customer-support routing data. Expect a drop on other domains. The external benchmark result (65.55%) shows this.
2. Only choice questions were trained. The noul (yes/no) and score (level) heads were not trained or calibrated on this data (calibration reported fitted_types: [0], choice only). The example above works, but treat these two types as unvalidated and test them on your own data.
3. Synthetic data. Real tickets are messier: typos, sarcasm, multiple intents, mixed languages. Performance on real traffic is not measured here.
4. Short training. One epoch of 300 steps. More data, epochs or a larger context window would likely help.
5. Possible train/held-out overlap. Rows were sampled at random and the notebook does not confirm that conversation groups (group_id) were kept on one side of the split, so the held-out score may be inflated.
6. Calibration is not a guarantee. On the external set, ECE was 0.132. Do not treat probabilities as true likelihoods on out-of-domain inputs.

Bias, privacy and safety

• The training conversations are synthetic. Customer identifiers look like Harbor-65673ba098 and are generated, not real people.
• The model has not been audited for fairness across dialects, demographics or products. Validate before using it on decisions that affect people.
• Use probability thresholds and keep a human review path for low-confidence cases.

Licensing

• Model weights: inherit the license of the base model (unsloth/Qwen3.5-4B). Check it before commercial use.
• Training data: the generated conversations are CC0-1.0. The upstream question descriptions are noted in the dataset as based on TypeSafe documentation with license not verified.

Reproduce

The full training notebook is Qwen3_5 (4B) Decision.ipynb. Pipeline: FastDecisionModel.from_pretrained → get_peft_model → build_dataset → split_holdout → DecisionTrainer → calibrate → evaluate → push_to_hub_merged.

Citation

@misc{jwen_decision_2026,
  author = {nikhil061307},
  title  = {Jwen Decision: a calibrated decision model fine-tuned from Qwen3.5-4B},
  year   = {2026},
  url    = {https://huggingface.co/nikhil061307/jwen_decision}
}

Acknowledgements

Built with Unsloth on Qwen3.5-4B. Data from Open-Jev / typed-decisions.