Qwen3.5-9B-FP8-Dynamic-LLMCompressor

ๅŸบไบŽ Qwen3.5-9B ๅˆถไฝœ็š„ ๅŠจๆ€ FP8 ้‡ๅŒ–ๆจกๅž‹๏ผŒไฝฟ็”จ llm-compressor 0.14.0 ๅˆถไฝœ๏ผŒไปฅ compressed-tensors 0.19.0 ๆ ผๅผไฟๅญ˜ใ€‚

้‡ๅŒ–้…็ฝฎ

้‡‡็”จ FP8_DYNAMIC ๅฏน 128 ไธช่ฏญ่จ€็บฟๆ€งๅฑ‚ๅš้‡ๅŒ–ใ€‚ๆƒ้‡ๆŒ‰่พ“ๅ‡บ้€š้“ไฝฟ็”จ้™ๆ€ๅฐบๅบฆ๏ผŒๆฟ€ๆดปๆŒ‰ token ๅŠจๆ€็ผฉๆ”พใ€‚ๆƒ้‡ๅ…ฑ 13.045 GiB๏ผŒๅŒ…ๆ‹ฌ 7 ไธชไธปไฝ“ๅˆ†็‰‡ๅ’Œ 1 ไธช MTP ๅˆ†็‰‡ใ€‚ visualใ€linear_attnใ€lm_head ๅŠๅ…ถไป–้ž็›ฎๆ ‡ๅฑ‚ไฟ็•™ๆตฎ็‚น็ฒพๅบฆ๏ผŒKV cache ๆฒฟ็”จๆŽจ็†ๅผ•ๆ“Ž็š„้ป˜่ฎค่ฎพ็ฝฎใ€‚

้‡ๅปบไฝฟ็”จไธŠๆธธๆไบค 460979c3d11864dd16408d860ac930a360a2fac2ใ€‚ๅŸบๅบงๆ–‡ไปถๆ ก้ชŒ่ฎฐๅฝ•่ง BASE_PROVENANCE.jsonใ€‚15 ไธช MTP ๅผ ้‡ไธŽๅŸบๅบง้€ๅญ—่Š‚ไธ€่‡ดใ€‚

ๆ–‡ๆœฌๆŽจ็†

ๅฎŒๆ•ดไธ‹่ฝฝไป“ๅบ“ๅŽ๏ผŒๅœจๆ”ฏๆŒ compressed-tensors ็š„ vLLM 0.32.0 ็Žฏๅขƒ่ฟ่กŒใ€‚ๆจกๅž‹็›ฎๅฝ•ๅบ”ๅŒ…ๅซๆƒ้‡ใ€็ดขๅผ•ใ€้…็ฝฎๅ’Œ tokenizerใ€‚็คบไพ‹ไฝฟ็”จๆ–‡ๆœฌ่พ“ๅ…ฅใ€‚

from vllm import LLM, SamplingParams

llm = LLM(
    model="./Qwen3.5-9B-FP8-Dynamic-LLMCompressor",
    dtype="float16", enforce_eager=True,
    max_model_len=4096, max_num_seqs=1,
    gpu_memory_utilization=0.90,
    enable_prefix_caching=False,
    limit_mm_per_prompt={"image": 0, "video": 0},
    generation_config="vllm",
)
tokenizer = llm.get_tokenizer()
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "็”จไธ€ๅฅ่ฏ่งฃ้‡ŠไบŒๅˆ†ๆŸฅๆ‰พใ€‚"}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
result = llm.generate([prompt], SamplingParams(temperature=0, max_tokens=192))
print(result[0].outputs[0].text)

้ชŒ่ฏ็ป“ๆžœ

ๅœจ RTX 4090 D ไธŠ้€š่ฟ‡ Transformers FP16 ่งฃๅŽ‹้‡่ฝฝใ€‚vLLM 0.32.0 ้€‰็”จ CUTLASS FP8 ๅ†…ๆ ธ๏ผŒ8 ๆกๆ–‡ๆœฌๅŠŸ่ƒฝๆ ทไพ‹้€š่ฟ‡ใ€‚ๅฝ“ๅ‰้‡ๅปบ็‰ˆ็š„้ชŒ่ฏ่Œƒๅ›ดไธบ็Ÿญๆ–‡ๆœฌ็”Ÿๆˆใ€‚

ๅค็ŽฐไธŽ่ฎธๅฏ

reproducibility/ ไฟๅญ˜็Žฏๅขƒใ€็›ฎๆ ‡ๆจกๅ—ๆธ…ๅ•ๅ’Œ้‡ๅŒ–่„šๆœฌใ€‚้‡ๅปบ็‰ˆๅ‘ๅธƒ่ฎฐๅฝ•่ง RELEASE_STATUS.json๏ผŒๆ—ง็‰ˆ่ฎฐๅฝ•่ง history/ใ€‚

้‡ๅŒ–ๅ‚ๆ•ฐ่ง config.json ๅ’Œ recipe.yaml๏ผŒๆ–‡ไปถๆ ก้ชŒๅ€ผ่ง SHA256SUMSใ€‚ๆจกๅž‹ๆฒฟ็”จๅŸบๅบง็š„ Apache-2.0 ่ฎธๅฏใ€‚

ไธ‹่ฝฝ

ไธŽไฝœ่€…็š„ ModelScope ไป“ๅบ“ๅŒๆญฅๅ‘ๅธƒ๏ผŒๆƒ้‡ใ€ๆ•ฐๆฎๅŠ่ฏ„ๆต‹่ฎฐๅฝ•ไฟๆŒไธ€่‡ดใ€‚

pip install -U huggingface_hub
hf download wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor --local-dir ./Qwen3.5-9B-FP8-Dynamic-LLMCompressor

ไธ‹่ฝฝๅŽ่ฟ›ๅ…ฅไป“ๅบ“็›ฎๅฝ•๏ผŒๅ†ๆŒ‰ไธŠ้ข็š„็คบไพ‹่ฟ่กŒใ€‚

Downloads last month
49
Safetensors
Model size
10B params
Tensor type
F16
ยท
F8_E4M3
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor

Finetuned
Qwen/Qwen3.5-9B
Quantized
(742)
this model