Instructions to use iromu/Qwen3-0.6B-tools with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use iromu/Qwen3-0.6B-tools with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="iromu/Qwen3-0.6B-tools") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("iromu/Qwen3-0.6B-tools") model = AutoModelForCausalLM.from_pretrained("iromu/Qwen3-0.6B-tools", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use iromu/Qwen3-0.6B-tools with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "iromu/Qwen3-0.6B-tools" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "iromu/Qwen3-0.6B-tools", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/iromu/Qwen3-0.6B-tools
- SGLang
How to use iromu/Qwen3-0.6B-tools with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "iromu/Qwen3-0.6B-tools" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "iromu/Qwen3-0.6B-tools", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "iromu/Qwen3-0.6B-tools" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "iromu/Qwen3-0.6B-tools", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use iromu/Qwen3-0.6B-tools with Docker Model Runner:
docker model run hf.co/iromu/Qwen3-0.6B-tools
Qwen3-0.6B-tools
Merged BF16 weights of the LoRA tool-calling fine-tune of Qwen/Qwen3-0.6B,
trained so the model reliably emits OpenAI-style tool calls for agentic
workflows.
Base model
This model was fine-tuned from:
Qwen/Qwen3-0.6B
Training
Supervised fine-tuning with LoRA, run with NeMo-RL, assistant-turn-masked cross-entropy over the tool-calling logs.
- LoRA: dim
32, alpha32, dropout0.05, target modules*.proj(q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj) - Max sequence length:
4096 - Precision:
bfloat16 - Optimizer: AdamW, lr
5e-5, weight decay0.01 - Schedule: 15-step linear warmup, then cosine decay to
1e-6 - Global batch size:
64(micro batch2) - Training steps:
336(validation every84) - Loss masked to assistant turns via the chat template's
generationkeyword
The tool-calling chat template is the model's contract: it is the template the
base model's inline template was replaced with so that tool schemas render
verbatim and the assistant's tool_calls block is the trainable span.
Dataset
Training used the sft_tools split of the
r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation dataset (OpenAI messages
format; assistant turns carry tool_calls).
Intended use
- Structured tool / function calling
- Agent-style multi-step interactions
- Small-footprint on-device or edge deployment
It is not intended to be a general replacement for larger Qwen models.
Sibling artifacts
- GGUF quantizations:
iromu/Qwen3-0.6B-tools-GGUF - NVFP4 checkpoint:
iromu/Qwen3-0.6B-tools-NVFP4
Usage
Serve the NVFP4 checkpoint with TensorRT-LLM:
trtllm-serve serve iromu/Qwen3-0.6B-tools-NVFP4 --port 8000
Load a GGUF quant with llama.cpp:
llama-cli -hf iromu/Qwen3-0.6B-tools-GGUF:Q4_K_M
Load the merged BF16 model with transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"iromu/Qwen3-0.6B-tools", torch_dtype="bfloat16", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("iromu/Qwen3-0.6B-tools")
Validation
| Model | Quant | Cap | n | Tool call emitted | Names match | Exact args match | Δ exact vs BASE (gated) |
|---|---|---|---|---|---|---|---|
| Qwen/Qwen3-0.6B | BF16 | 384 tok | 274 | 48/274 (17.5%) | 9/274 (3.3%) | 2/274 (0.7%) | (no predecessor) |
| Qwen3-0.6B-tools | BF16 | 384 tok | 274 | 272/274 (99.3%) | 228/274 (83.2%) | 175/274 (63.9%) | +63.1pp |
Measured on the sft_tools held-out validation split, greedy decoding. Throughput is single-stream greedy decode, not serving throughput.
suppressed in the Δ column means no comparison was admissible, which is a statement about the provenance of the two rows and not a measurement of no change. The reason is printed in the cell.
- Downloads last month
- 512