AbhishekG711/Qwen3-0.6B-Insight-Extractor

Model Summary

Qwen3-0.6B-Insight-Extractor is a LoRA fine-tune of Qwen/Qwen3-0.6B. It converts a raw customer-support ticket into a single structured JSON object across 9 fields, for automated ticket classification and downstream entity-extraction workflows.

Base model	Qwen/Qwen3-0.6B
Fine-tuning method	LoRA Adapters via Unsloth + TRL SFTTrainer
Task	Structured JSON extraction from support-ticket text
Training data	AbhishekG711/processed_support_tickets (train split)
Target repo	AbhishekG711/Qwen3-0.6B-Insight-Extractor

Base Model Architecture (Qwen3-0.6B)

Type	Dense, causal decoder-only Transformer
Parameters	0.6B total (~0.44B non-embedding)
Layers	28
Attention	Grouped-Query Attention — 16 query heads / 8 KV heads
Native context length	32,768 tokens
Reasoning modes	Hybrid — supports both "thinking" (<think>...</think>) and "non-thinking" modes via enable_thinking
Base model license	Apache 2.0

Intended Use

Given a raw support ticket, produce one JSON object for automated triage, routing, sentiment/urgency dashboards, or downstream alerting — not a conversational assistant.

Fine-Tuning Training Details

Hyperparameter	Value
Quantization	None (load_in_4bit=False)
LoRA rank (r)	32
LoRA alpha	64 (= 2 × r)
LoRA dropout	0.0
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
LoRA bias	none
Max sequence length	4096
Per-device train batch size	4
Target global batch size	32
Gradient accumulation steps	8 (= 32 // 4)
Epochs	1
Learning rate	2e-4
LR scheduler	linear
Warmup steps	10
Weight decay	0.01
Optimizer	adamw_8bit
Precision	bf16 if supported, else fp16 (auto-detected via is_bf16_supported())
Completion-only loss	True — loss computed only on completion tokens, not the prompt
Sequence packing	False
Eval strategy	every 25 steps
Save strategy	every 50 steps, save_total_limit=3
Load best model at end	True (metric_for_best_model="eval_loss", greater_is_better=False)
Gradient checkpointing	"unsloth" mode
Seed	3407
Experiment tracker	report_to="none" — console/log file only

Evaluation

Metric	Value
Validation loss (eval_loss, best checkpoint)	0.233829

How to Prompt This Model

The model was fine-tuned against exactly this system/user structure — matching it as closely as possible at inference time will give the best results.

System prompt:

# ROLE
You are a deterministic extraction engine. You convert raw support emails into exactly one structured JSON object. You never converse, explain, or output anything except that JSON object.

User message template:

Analyze the following customer support email and extract structured insights.

<raw ticket text>

Expected output: One JSON object with exactly these 9 keys, in this order: is_actionable, summary, sentiment, category, intent, aspect, urgency, reported_cause, entities.

Inference example

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AbhishekG711/Qwen3-0.6B-Insight-Extractor"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

SYSTEM_PROMPT = """
# ROLE
You are a deterministic extraction engine. You convert raw support emails into exactly one structured JSON object. You never converse, explain, or output anything except that JSON object.
"""
USER_INSTRUCTION = "Analyze the following customer support email and extract structured insights.\n\n"

ticket_text = "My order #4471 arrived damaged and support hasn't replied in 3 days."

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": USER_INSTRUCTION + ticket_text},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

output = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))