ctech-fle-gt / mtp_draft /plugin /q38_mtp_draft.py
ctech123's picture
Duplicate from ctech123/Qwen3.8-Flash-Next-E384-FP4-V2
d07994b
Raw History Blame Contribute Delete
807 Bytes
"""vLLM general plugin: install the cheap Qwen3.8-Flash-Next MTP drafter (q38_draft.py) in every vLLM process.
Inactive unless Q38_DRAFT_VOCAB is set, so the same environment still runs unpatched servers.
Q38_DRAFT_HOME directory holding q38_draft.py / nvfp4_ops.py (this repo's mtp_draft/)
Q38_DRAFT_VOCAB draft vocabulary mask (.pt bool[248320]); empty -> plugin inactive
Q38_DRAFT_HEAD_Q marlin (default) | cutlass | bf16 draft lm_head
Q38_DRAFT_MTP_Q marlin (default) | cutlass | bf16 MTP block BF16 linears
"""
import os
import sys
def main():
if not os.environ.get("Q38_DRAFT_VOCAB"):
return
home = os.environ.get("Q38_DRAFT_HOME")
if home and home not in sys.path:
sys.path.insert(0, home)
import q38_draft
q38_draft.install_from_env()