RedHatAI/Qwen3.5-397B-A17B-speculator.dflash

This is a DFlash speculator model for Qwen/Qwen3.5-397B-A17B.

Training Details

This model was trained using the Speculators library on a subset of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and the train_sft split of HuggingFaceH4/ultrachat_200k. Responses were regenerated by RedHatAI/Qwen3.5-397B-A17B and stored at Qwen3.5-397B-A17B-responses

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3.5-397B-A17B
  --data ./regenerated_data.jsonl \
  --assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
  --output ./output \
  --seq-length 16384

Launch vLLM

# In (separate) virtual environment with [vLLM](https://github.com/imargulis/vllm/tree/fix/extract-hidden-states-hybrid-block-size) installed
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \
  Qwen/Qwen3.5-397B-A17B \
  --target-layer-ids 3 15 23 35 47 \
  --max-model-len  32000 \
  --reasoning-parser qwen3 \
  --language-model-only \
  --kv-cache-dtype fp8 \ 
  --max-num-batched-tokens 32768\
  --tensor-parallel-size 4 \ 
  --async-scheduling

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3.5-397B-A17B \     
  --data-path ./output \    
  --on-missing generate \    
  --on-generate delete \    
  --scheduler-type cosine \    
  --draft-vocab-size 24576 \    
  --max-anchors 3072 \    
  --target-layer-ids 3 15 23 35 47 \
  --speculator-type dflash \     
  --logger trackio  \    
  --lr 0.0006 \    
  --epochs 5 \ 
  --draft-config 397_config.json

Model Specifications


Base Model	Qwen/Qwen3.5-397B-A17B
Chat Template	Qwen/Qwen3.5-397B-A17B (use `/chat/completions` endpoint)
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia H200

Deployment

# Install vLLM from the required PR
pip install git+https://github.com/vllm-project/vllm.git  
                                                                                                                                                                                                                                                                                                          
# Deploy with speculative decoding                                                                                                                                                                                                                                                                        
vllm serve Qwen/Qwen3.5-397B-A17B \                                                                                                                                                                                                                                                                                
    --tensor-parallel-size 8 \                                                                                                                                                                                                                                                                            
    --max-num-batched-tokens 32768 \
    --attention-backend FLASH_ATTN \ 
    --speculative-config '{                                                                                                                                                                                                                                                                               
        "model": "RedHatAI/Qwen3.5-397B-A17B-speculator.dflash",                                                                                                                                                                                                                                                   
        "num_speculative_tokens": 7,                                                                                                                                                                                                                                                                   
        "method": "dflash"                                                                                                                                                                                                                                                                                
    }'

Acceptance Rates

Per-position token acceptance rates across datasets:
(with reasoning enabled)

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Avg. Length
HumanEval	83.9%	68.6%	56.6%	46.9%	39.1%	32.4%	26.2%	4.54
math_reasoning	85.5%	71.9%	61.1%	52.3%	44.6%	37.6%	31.2%	4.84
qa	69.8%	47.3%	33.1%	24.5%	18.8%	14.0%	10.1%	3.18
question	73.6%	52.0%	37.6%	28.6%	22.7%	18.0%	13.9%	3.46
rag	78.7%	59.3%	45.0%	34.7%	26.9%	20.7%	15.6%	3.81
summarization	72.5%	49.6%	34.8%	25.5%	19.1%	14.4%	10.6%	3.27
tool_call	73.9%	52.2%	36.7%	26.2%	18.8%	13.3%	9.7%	3.31
translation	67.1%	41.3%	23.6%	13.0%	7.0%	3.8%	2.0%	2.58
writing	73.6%	52.0%	37.6%	28.6%	22.6%	18.0%	13.9%	3.46