Instructions to use wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor") model = AutoModelForMultimodalLM.from_pretrained("wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor
- SGLang
How to use wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor with Docker Model Runner:
docker model run hf.co/wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor
Qwen3.5-9B-FP8-Dynamic-LLMCompressor
ๅบไบ Qwen3.5-9B ๅถไฝ็ ๅจๆ FP8 ้ๅๆจกๅ๏ผไฝฟ็จ llm-compressor 0.14.0 ๅถไฝ๏ผไปฅ compressed-tensors 0.19.0 ๆ ผๅผไฟๅญใ
้ๅ้ ็ฝฎ
้็จ FP8_DYNAMIC ๅฏน 128 ไธช่ฏญ่จ็บฟๆงๅฑๅ้ๅใๆ้ๆ่พๅบ้้ไฝฟ็จ้ๆๅฐบๅบฆ๏ผๆฟๆดปๆ token ๅจๆ็ผฉๆพใๆ้ๅ
ฑ 13.045 GiB๏ผๅ
ๆฌ 7 ไธชไธปไฝๅ็ๅ 1 ไธช MTP ๅ็ใ visualใlinear_attnใlm_head ๅๅ
ถไป้็ฎๆ ๅฑไฟ็ๆตฎ็น็ฒพๅบฆ๏ผKV cache ๆฒฟ็จๆจ็ๅผๆ็้ป่ฎค่ฎพ็ฝฎใ
้ๅปบไฝฟ็จไธๆธธๆไบค 460979c3d11864dd16408d860ac930a360a2fac2ใๅบๅบงๆไปถๆ ก้ช่ฎฐๅฝ่ง BASE_PROVENANCE.jsonใ15 ไธช MTP ๅผ ้ไธๅบๅบง้ๅญ่ไธ่ดใ
ๆๆฌๆจ็
ๅฎๆดไธ่ฝฝไปๅบๅ๏ผๅจๆฏๆ compressed-tensors ็ vLLM 0.32.0 ็ฏๅข่ฟ่กใๆจกๅ็ฎๅฝๅบๅ ๅซๆ้ใ็ดขๅผใ้ ็ฝฎๅ tokenizerใ็คบไพไฝฟ็จๆๆฌ่พๅ ฅใ
from vllm import LLM, SamplingParams
llm = LLM(
model="./Qwen3.5-9B-FP8-Dynamic-LLMCompressor",
dtype="float16", enforce_eager=True,
max_model_len=4096, max_num_seqs=1,
gpu_memory_utilization=0.90,
enable_prefix_caching=False,
limit_mm_per_prompt={"image": 0, "video": 0},
generation_config="vllm",
)
tokenizer = llm.get_tokenizer()
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": "็จไธๅฅ่ฏ่งฃ้ไบๅๆฅๆพใ"}],
tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
result = llm.generate([prompt], SamplingParams(temperature=0, max_tokens=192))
print(result[0].outputs[0].text)
้ช่ฏ็ปๆ
ๅจ RTX 4090 D ไธ้่ฟ Transformers FP16 ่งฃๅ้่ฝฝใvLLM 0.32.0 ้็จ CUTLASS FP8 ๅ ๆ ธ๏ผ8 ๆกๆๆฌๅ่ฝๆ ทไพ้่ฟใๅฝๅ้ๅปบ็็้ช่ฏ่ๅดไธบ็ญๆๆฌ็ๆใ
ๅค็ฐไธ่ฎธๅฏ
reproducibility/ ไฟๅญ็ฏๅขใ็ฎๆ ๆจกๅๆธ ๅๅ้ๅ่ๆฌใ้ๅปบ็ๅๅธ่ฎฐๅฝ่ง RELEASE_STATUS.json๏ผๆง็่ฎฐๅฝ่ง history/ใ
้ๅๅๆฐ่ง config.json ๅ recipe.yaml๏ผๆไปถๆ ก้ชๅผ่ง SHA256SUMSใๆจกๅๆฒฟ็จๅบๅบง็ Apache-2.0 ่ฎธๅฏใ
ไธ่ฝฝ
ไธไฝ่ ็ ModelScope ไปๅบๅๆญฅๅๅธ๏ผๆ้ใๆฐๆฎๅ่ฏๆต่ฎฐๅฝไฟๆไธ่ดใ
pip install -U huggingface_hub
hf download wkhaifor/Qwen3.5-9B-FP8-Dynamic-LLMCompressor --local-dir ./Qwen3.5-9B-FP8-Dynamic-LLMCompressor
ไธ่ฝฝๅ่ฟๅ ฅไปๅบ็ฎๅฝ๏ผๅๆไธ้ข็็คบไพ่ฟ่กใ
- Downloads last month
- 49