ningpy/qwen2.5-7b-instruct-intent-v14

Qwen2.5-7B-Instruct Intent Classifier v14

Health-domain intent classifier fine-tuned from Qwen/Qwen2.5-7B-Instruct on 27,659 labeled samples.

v14 vs v13 (main improvements)

• Base model fix: v13 used Qwen2.5-7B (base) with eos_token = <|endoftext|> mismatched against the ChatML <|im_end|> used in training data — v13 model kept generating past its first JSON. v14 uses Qwen2.5-7B-Instruct with eos_token = <|im_end|> matching the training format.
• Parse error rate: 0.00% (v13: 0.12%)
• EOS close rate: 100.00% — every generation stops cleanly on <|im_end|>
• Train/eval loss: 0.026 / 0.023 (v13: 0.244 / 0.259) — 10× better convergence

Evaluation (n=2518 val)

Metric	Value
Intent accuracy	91.26%
State accuracy	85.86%
RedFlag accuracy	97.10%
block_code accuracy	75.34%
Joint accuracy	70.25%
Parse error rate	0.00%
EOS close rate	100.00%

Per-state F1

State	Precision	Recall	F1	Support
PASS	0.872	0.917	0.894	1014
BLOCK	0.887	0.869	0.878	858
STEER	0.672	0.597	0.632	196
REDFLAG	0.87	0.87	0.87	307
COEXIST	0.785	0.713	0.747	143

Per-intent F1

Intent	Precision	Recall	F1	Support
HA (health advisor)	0.922	0.941	0.931	1076
HCN (care nav)	0.859	0.931	0.893	144
MT (med tracking)	0.965	0.909	0.936	241
PI (pharmacy)	0.931	0.905	0.918	179
OTHER	0.888	0.878	0.883	698

Schema (flat 6-key JSON)

{
  "intent": "HA|HCN|MT|PI|OTHER",
  "redflag": true|false,
  "redflag_code": "",
  "block": "PASS|BLOCK_HARD|STEER_SOFT|NONE",
  "block_code": "PASS_*|OOS_H_*|OOS_S_*",
  "entities": {"drugs":[],"diseases":[],"symptoms":[]}
}

5 legal states (redflag, block, block_code):

• PASS : false, "PASS", PASS_*
• REDFLAG : true, "NONE", ""
• BLOCK : false, "BLOCK_HARD", OOS_H_*
• STEER : false, "STEER_SOFT", OOS_S_*
• COEXIST : true, "BLOCK_HARD", OOS_H_* (emergency + personal med action)

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from huggingface_hub import hf_hub_download
import torch

model_id = "ningpy/qwen2.5-7b-instruct-intent-v14"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="cuda")

SYS = open(hf_hub_download(model_id, "system_prompt_v14.txt")).read().strip()
msgs = [
    {"role": "system", "content": SYS},
    {"role": "user", "content": "Message: Metoclopramide 我该怎么服用这种药?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=250, do_sample=False, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# {"intent": "HA", "redflag": false, "redflag_code": "", "block": "PASS",
#  "block_code": "PASS_HA_USAGE",
#  "entities": {"drugs": ["Metoclopramide"], "diseases": [], "symptoms": []}}

Training

• Base: Qwen/Qwen2.5-7B-Instruct
• Method: LoRA r=32, alpha=64
• 3 epochs, cosine LR 1e-4 → 0, batch size 48
• 4×V100-32GB, 6h38m
• Final train loss 0.026 · Eval loss (in-training) 0.023

Data (27,659 train / 2,518 val):

• v12 → flat 5-state schema conversion
• +500 synthesized PASS_HA_USAGE seeds for <drug> + generic administration/timing templates
• GPT-4o relabel of 19 suspect BLOCK rows → 2 flipped to PASS