finetuning-qwen9b: Experiment 2 merged vLLM deployment

This repository contains the merged BF16 Experiment 2 model and its tokenizer/configuration files. It is prepared for direct vLLM deployment; no PEFT adapter is loaded at runtime.

Provenance

Item	Value
Base model	unsloth/Qwen3.5-9B
Training method	BF16 LoRA, rank 16, alpha 32
Selected checkpoint	checkpoint-750
Deployment tag	exp02-merged-vllm-v1
Deployment snapshot	42e426238b2cd0e333a65ed785addee107144f1b
Model format	Merged BF16 Safetensors, four shards
Total weight bytes	19,306,216,416
Intended task	Text function calling and agentic tool selection

The LoRA update has already been folded into the base weights. Do not use --enable-lora for this snapshot.

The API alias qwen9b-function-call is intentionally independent of the experiment number. LiteLLM and other clients can retain this model name while the server's pinned Hub revision changes during a controlled promotion.

Evaluation summary

Split	Composite	Exact tool set	Tool + arguments exact	Decision accuracy	Unauthorized writes	Forbidden tools
Validation	78.63%	81.96%	80.65%	98.04%	0%	0%
Test	79.12%	83.01%	82.37%	99.14%	0%	0%
Guard	84.59%	82.97%	81.66%	98.69%	0%	0%

Runtime authorization and confirmation remain responsibilities of the agent harness.

Coolify: replace Gemma on GPU 0

Use the official vllm/vllm-openai:v0.26.0 image. Store a fine-grained read-only token in the Coolify secret HF_TOKEN, preserve the Hugging Face cache, stop the old Gemma service, and deploy:

services:
  vllm_finetuned_qwen9b:
    image: vllm/vllm-openai:v0.26.0
    restart: unless-stopped
    init: true
    ipc: host
    command:
      - --model
      - Koyiljon/finetuning-qwen9b
      - --revision
      - 42e426238b2cd0e333a65ed785addee107144f1b
      - --served-model-name
      - qwen9b-function-call
      - --tensor-parallel-size
      - "1"
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --dtype
      - bfloat16
      - --max-num-seqs
      - "18"
      - --max-model-len
      - "4096"
      - --max-num-batched-tokens
      - "4096"
      - --gpu-memory-utilization
      - "0.85"
      - --enable-prefix-caching
      - --language-model-only
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_coder
      - --default-chat-template-kwargs
      - '{"enable_thinking":false}'
      - --download-dir
      - /root/.cache/huggingface
    ports:
      - "8091:8000"
    volumes:
      - /opt/huggingface_cache_vllm:/root/.cache/huggingface
    environment:
      HF_HOME: /root/.cache/huggingface
      HF_TOKEN: "${HF_TOKEN:?Add a fine-grained read token for Koyiljon/finetuning-qwen9b}"
      HUGGING_FACE_HUB_TOKEN: "${HF_TOKEN:?Add a fine-grained read token for Koyiljon/finetuning-qwen9b}"
      NVIDIA_DRIVER_CAPABILITIES: compute,utility
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]
    healthcheck:
      test:
        - CMD
        - python3
        - -c
        - "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3)"
      interval: 60s
      timeout: 5s
      retries: 5
      start_period: 600s

The official image already supplies the vLLM server entrypoint. Do not prepend vllm serve, and do not reuse Gemma's audio installation shell command.

This service reserves host GPU 0 and host port 8091. It does not alter the separate Qwen3.6-35B service on GPU 1 and port 8090.

Docker

docker run --rm --runtime nvidia --gpus '"device=0"' \
  --ipc=host \
  -p 8091:8000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v /opt/huggingface_cache_vllm:/root/.cache/huggingface \
  vllm/vllm-openai:v0.26.0 \
  --model Koyiljon/finetuning-qwen9b \
  --revision 42e426238b2cd0e333a65ed785addee107144f1b \
  --served-model-name qwen9b-function-call \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --max-num-seqs 18 \
  --max-model-len 4096 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --language-model-only \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking":false}'

Smoke checks

curl -fsS http://127.0.0.1:8091/health
curl -fsS http://127.0.0.1:8091/v1/models

The model list must contain qwen9b-function-call. For a tool-call request, use temperature=0, max_tokens=512, and initially stream=false. Confirm the response exposes calls through choices[0].message.tool_calls. Then test the actual streaming path and run the protected function-calling and guard evaluation suites before production promotion.

Important notes

• This is a BF16 artifact, not FP8, INT8, or QLoRA.
• --language-model-only is intentional because this fine-tune targets text function calling.
• Keep the endpoint private or protect it with authentication.
• A healthy endpoint does not prove prompt, EOS, parser, or metric parity.