Text Generation
Safetensors
English
qwen2
gptqmodel
modelcloud
code
codeqwen
chat
qwen
qwen-coder
instruct
int4
gptq
4bit
conversational
4-bit precision
Instructions to use ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Local Apps Settings
- vLLM
How to use ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1
- SGLang
How to use ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1 with Docker Model Runner:
docker model run hf.co/ModelCloud/Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1
Commit History
-
Upload folder using huggingface_hub (#3)
Verified signature- e8567040d4a4206b13a8a501bd7f59c78fa5cd5109dae64383cfb1db73610cbf (dfe42b890d8ea51c536ea5b9f977a102100b9edf) - 731b4cdacd3b30f051103012ac3f9b7e6388237261e2ddabd5948f62f7e00eb6 (619c731aeaceb80705d9bcee5e5f1dcf538045c2) - c95f793f89216c6a3ffbd8842c305185090cf8ed74f11f21a8984af6b7d9008f (0f65e6e263e0c61820f0a2fc0a507f004a04517c) - 537e527a138486e6b9201f385d34c679d14eb968a0ed8dd8d972176e1a6943b9 (240605ec337e64bb45454a3f4e640949d95e4359) - 4df5a7a0bd69535850cffe72de19325d4055502e3e17d0e8d428fb72f63df45e (dc681348cf7c6cc090724a34b21b2e5ea107c0f3) - 022ad0d97a033fa0cd70bfa00f177b7490843657f17207814a4b8888b1be0c14 (2c8a01187b0365e17c11da9016df073ab13c9d8f)61e6721
-
Upload folder using huggingface_hub (#1)
Verified signature- f9689eb2e7de123ec5e6c9aaa375e040cc143a2c0c52ecdaf4b8412bb66519fc (330b1f7e9fc6b864ca45fc7c8539a8a274db5595) - 6ea60db1f8c653239860986e16ac0a2e9ab248e2df22f505175a724f16147fb0 (8ad6f77daaa3c8a98c4a75959938d9d2755910d9)7c6b254