Instructions to use JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2")
model = AutoModelForCausalLM.from_pretrained("JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2

SGLang

How to use JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2 with Docker Model Runner:
```
docker model run hf.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2
```

JaspinXu commited on 8 days ago

Commit

7f8c3dc

verified ·

1 Parent(s): 9964a8d

Update model card

Browse files

Files changed (1) hide show

README.md +59 -44

README.md CHANGED Viewed

@@ -25,41 +25,83 @@ tags:
 **Code** [github.com/JaspinXu/KnowWhenToHandOff](https://github.com/JaspinXu/KnowWhenToHandOff) · **Data** [KnowWhenToHandOff-data](https://huggingface.co/datasets/JaspinXu/KnowWhenToHandOff-data) · **Family** [SFT](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-SFT) · [R1](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R1) · [R2](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2)
 ## Quick start
-The model is a standard Qwen3 chat model with tool calling. It was trained with
-the τ²-bench domain policy as the system prompt and the domain's tools,
-including `transfer_to_human_agents`; for faithful behaviour run it inside the
-evaluation harness of the [code repository](https://github.com/JaspinXu/KnowWhenToHandOff).
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 repo = "JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2"
 tokenizer = AutoTokenizer.from_pretrained(repo)
-model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
-tools = [{"type": "function", "function": {
-    "name": "transfer_to_human_agents",
-    "description": "Transfer the user to a human agent with a summary of the issue.",
-    "parameters": {"type": "object", "properties": {"summary": {"type": "string"}}, "required": ["summary"]},
-}}]
 messages = [
-    {"role": "system", "content": "<the airline or retail policy from tau2-bench>"},
-    {"role": "user", "content": "I need to book the unaccompanied-minor escort service for my child."},
 ]
-inputs = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_tensors="pt").to(model.device)
 output = model.generate(inputs, max_new_tokens=512)
 print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))
 ```
-Serving with vLLM (OpenAI-compatible, tool calls parsed):
 ```bash
-vllm serve JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2 --enable-auto-tool-choice --tool-call-parser hermes
 ```
-## Model
 - Base model: Qwen/Qwen3-4B-Instruct-2507 (revision
   `cdbee75f17c01a7cc42f958dc650907174af0554`), Apache-2.0.
@@ -70,17 +112,6 @@ vllm serve JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2 --enable-auto-tool-choice --to
   `fc0055dc4e0a316c3f83133267fbd6faaa770992`; all sources in
   [reports/main/sources.json](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/sources.json).
-## Model family
-| Model | Training | Hand-off F1 | Over-escalation | Use it for |
-| --- | --- | --- | --- | --- |
-| [SFT (B2)](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-SFT) | LoRA SFT on 790 filtered teacher dialogues | 0.768 | 0.016 | starting point; precise but misses hand-offs |
-| [R1](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R1) | GRPO from B2, task reward only | 0.802 | 0.462 | studying the outcome-reward loophole |
-| [R2](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2) | GRPO from B2, task reward + violation and hand-off penalties | **0.812** | 0.103 | the best-calibrated hand-off model |
-Task success on the official test tasks is the same for all three within
-noise (0.33–0.35).
 ## Intended use
 Studying when a tool-using customer-service agent should hand off to a human,
@@ -95,22 +126,6 @@ requests for a human, out-of-scope requests, hard negatives); the user
 simulator is Qwen3-30B-A3B-Instruct-2507. No real user data. Test tasks were
 never used for training or selection.
-## Evaluation
-Test split, protocol frozen in [ADR-005](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/docs/adr/ADR-005-evaluation-protocol.md); 95 % bootstrap intervals over tasks.
-| Metric | This model | B2 (SFT start) | Source |
-| --- | --- | --- | --- |
-| Success (pass^1), test_main | 0.342 [0.245, 0.449] | 0.327 [0.224, 0.434] | [main table](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/main_table.md) |
-| Paired Δ success vs B2 | +0.015 [−0.056, 0.082] | — | [paired differences](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/paired_differences.md) |
-| pass^4 | 0.122 [0.041, 0.224] | 0.102 [0.041, 0.204] | [main table](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/main_table.md) |
-| Violation rate (incl. off-reference writes) | 0.444 | 0.469 | same |
-| Hand-off F1 / precision / recall, test_handoff | **0.812** / 0.886 / 0.750 | 0.768 / 0.976 / 0.633 | [`R2-final-test_handoff-s0-20261007`](https://huggingface.co/datasets/JaspinXu/KnowWhenToHandOff-data/blob/main/runs/R2-final-test_handoff-s0-20261007/metrics.json) |
-| Over-escalation, test_handoff | 0.103 | 0.016 | same |
-| Worst-style success / style gap | 0.316 / 0.112 | 0.327 / 0.112 | [styles](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/styles.md) |
-**Known behaviour.** R2 removes R1's over-escalation but sometimes makes the opposite error on out-of-scope requests: it refuses and tells the user to contact the company directly without calling the transfer tool, or substitutes an unrelated write (under-escalation 0.25). Once it justified a hand-off with a policy that does not exist.
 ## Limitations
 - One user simulator (Qwen3-30B-A3B-Instruct-2507, which also generated the

 **Code** [github.com/JaspinXu/KnowWhenToHandOff](https://github.com/JaspinXu/KnowWhenToHandOff) · **Data** [KnowWhenToHandOff-data](https://huggingface.co/datasets/JaspinXu/KnowWhenToHandOff-data) · **Family** [SFT](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-SFT) · [R1](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R1) · [R2](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2)
+**R2** is Qwen3-4B-Instruct-2507 trained with SFT and then multi-turn GRPO on the τ²-bench airline and retail domains, with a reward that adds penalties for rule violations, unnecessary hand-offs and missed hand-offs. It has the best hand-off F1 of the study (0.812) with 10 % over-escalation, at the same task success as its SFT starting point.
+## Model family
+| Model | Training | Hand-off F1 | Over-escalation | Use it for |
+| --- | --- | --- | --- | --- |
+| [SFT (B2)](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-SFT) | LoRA SFT on 790 filtered teacher dialogues | 0.768 | 0.016 | starting point; precise but misses hand-offs |
+| [R1](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R1) | GRPO from B2, task reward only | 0.802 | 0.462 | studying the outcome-reward loophole |
+| [R2](https://huggingface.co/JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2) | GRPO from B2, task reward + violation and hand-off penalties | **0.812** | 0.103 | the best-calibrated hand-off model |
+Task success on the official test tasks is the same for all three within
+noise (0.33–0.35).
 ## Quick start
+A standard Qwen3 chat model with tool calling. It was trained with the
+τ²-bench domain policy as the system prompt and the domain's tools, including
+`transfer_to_human_agents`; for faithful behaviour run it inside the evaluation
+harness of the [code repository](https://github.com/JaspinXu/KnowWhenToHandOff).
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 repo = "JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2"
 tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(
+    repo, torch_dtype="auto", device_map="auto"
+)
+transfer = {
+    "type": "function",
+    "function": {
+        "name": "transfer_to_human_agents",
+        "description": "Transfer the user to a human agent.",
+        "parameters": {
+            "type": "object",
+            "properties": {"summary": {"type": "string"}},
+            "required": ["summary"],
+        },
+    },
+}
 messages = [
+    {"role": "system", "content": "<airline or retail policy>"},
+    {"role": "user", "content": "I need the unaccompanied-minor service."},
 ]
+inputs = tokenizer.apply_chat_template(
+    messages, tools=[transfer], add_generation_prompt=True,
+    return_tensors="pt",
+).to(model.device)
 output = model.generate(inputs, max_new_tokens=512)
 print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))
 ```
+Serve it with vLLM (OpenAI-compatible API, tool calls parsed):
 ```bash
+vllm serve JaspinXu/KnowWhenToHandOff-Qwen3-4B-R2 \
+  --enable-auto-tool-choice --tool-call-parser hermes
 ```
+## Evaluation
+Test split, protocol frozen in [ADR-005](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/docs/adr/ADR-005-evaluation-protocol.md); 95 % bootstrap intervals over tasks.
+| Metric | This model | B2 (SFT start) | Source |
+| --- | --- | --- | --- |
+| Success (pass^1), test_main | 0.342 [0.245, 0.449] | 0.327 [0.224, 0.434] | [main table](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/main_table.md) |
+| Paired Δ success vs B2 | +0.015 [−0.056, 0.082] | — | [paired differences](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/paired_differences.md) |
+| pass^4 | 0.122 [0.041, 0.224] | 0.102 [0.041, 0.204] | [main table](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/main_table.md) |
+| Violation rate (incl. off-reference writes) | 0.444 | 0.469 | same |
+| Hand-off F1 / precision / recall, test_handoff | **0.812** / 0.886 / 0.750 | 0.768 / 0.976 / 0.633 | [`R2-final-test_handoff-s0-20261007`](https://huggingface.co/datasets/JaspinXu/KnowWhenToHandOff-data/blob/main/runs/R2-final-test_handoff-s0-20261007/metrics.json) |
+| Over-escalation, test_handoff | 0.103 | 0.016 | same |
+| Worst-style success / style gap | 0.316 / 0.112 | 0.327 / 0.112 | [styles](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/styles.md) |
+**Known behaviour.** R2 removes R1's over-escalation but sometimes makes the opposite error on out-of-scope requests: it refuses and tells the user to contact the company directly without calling the transfer tool, or substitutes an unrelated write (under-escalation 0.25). Once it justified a hand-off with a policy that does not exist.
+## Training details
 - Base model: Qwen/Qwen3-4B-Instruct-2507 (revision
   `cdbee75f17c01a7cc42f958dc650907174af0554`), Apache-2.0.
   `fc0055dc4e0a316c3f83133267fbd6faaa770992`; all sources in
   [reports/main/sources.json](https://github.com/JaspinXu/KnowWhenToHandOff/blob/main/reports/main/sources.json).
 ## Intended use
 Studying when a tool-using customer-service agent should hand off to a human,
 simulator is Qwen3-30B-A3B-Instruct-2507. No real user data. Test tasks were
 never used for training or selection.
 ## Limitations
 - One user simulator (Qwen3-30B-A3B-Instruct-2507, which also generated the