Instructions to use AngelSlim/Hy3-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Local Apps Settings
- vLLM
How to use AngelSlim/Hy3-GPTQ-Int4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AngelSlim/Hy3-GPTQ-Int4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AngelSlim/Hy3-GPTQ-Int4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/AngelSlim/Hy3-GPTQ-Int4
- SGLang
How to use AngelSlim/Hy3-GPTQ-Int4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AngelSlim/Hy3-GPTQ-Int4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AngelSlim/Hy3-GPTQ-Int4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AngelSlim/Hy3-GPTQ-Int4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AngelSlim/Hy3-GPTQ-Int4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use AngelSlim/Hy3-GPTQ-Int4 with Docker Model Runner:
docker model run hf.co/AngelSlim/Hy3-GPTQ-Int4
Commit History
-
Update README.md
Verified signature5c81fda -
Update config.json
Verified signatureab22d6b -
add mtp
Verified signature7076356
-
Delete README_CN.md
Verified signaturede94184 -
Update README.md
Verified signature89ae728 -
Create README.md (#1)
- Create README.md (0e5677524ad860f2f295f5fc0ac105bb12ca3a61)143f1a9 -
Upload 2 files
Verified signature2a3793d -
Upload tokenizer.json with huggingface_hub
Verified signature27935e2 -
Upload tokenizer_config.json with huggingface_hub
Verified signature496ba74 -
Upload README.md with huggingface_hub
Verified signature4c483b6 -
Upload README_CN.md with huggingface_hub
Verified signatured43047f -
Upload model.safetensors.index.json with huggingface_hub
Verified signatureec8e680 -
Upload model-00034-of-00034.safetensors with huggingface_hub
Verified signature3cc0746 -
Upload model-00033-of-00034.safetensors with huggingface_hub
Verified signature98f94ec -
Upload model-00032-of-00034.safetensors with huggingface_hub
Verified signaturef7afd2e -
Upload model-00031-of-00034.safetensors with huggingface_hub
Verified signaturec2fa04f -
Upload model-00030-of-00034.safetensors with huggingface_hub
Verified signature96c8844 -
Upload model-00029-of-00034.safetensors with huggingface_hub
Verified signatureca53831 -
Upload model-00028-of-00034.safetensors with huggingface_hub
Verified signature5c014fe -
Upload model-00027-of-00034.safetensors with huggingface_hub
Verified signaturea324dcc -
Upload model-00026-of-00034.safetensors with huggingface_hub
Verified signature859342e -
Upload model-00025-of-00034.safetensors with huggingface_hub
Verified signaturead48db8 -
Upload model-00024-of-00034.safetensors with huggingface_hub
Verified signature3d81d91 -
Upload model-00023-of-00034.safetensors with huggingface_hub
Verified signature9556f2b -
Upload model-00022-of-00034.safetensors with huggingface_hub
Verified signaturefb5d41a -
Upload model-00021-of-00034.safetensors with huggingface_hub
Verified signaturefe3e9f2 -
Upload model-00020-of-00034.safetensors with huggingface_hub
Verified signaturee38e7f9 -
Upload model-00019-of-00034.safetensors with huggingface_hub
Verified signaturedb80287 -
Upload model-00018-of-00034.safetensors with huggingface_hub
Verified signature4a7c88a -
Upload model-00017-of-00034.safetensors with huggingface_hub
Verified signature7d22f88 -
Upload model-00016-of-00034.safetensors with huggingface_hub
Verified signature715d54c -
Upload model-00015-of-00034.safetensors with huggingface_hub
Verified signature7a5f603 -
Upload model-00014-of-00034.safetensors with huggingface_hub
Verified signature6d42a3f -
Upload model-00013-of-00034.safetensors with huggingface_hub
Verified signature9e509d7 -
Upload model-00012-of-00034.safetensors with huggingface_hub
Verified signaturece256f7 -
Upload model-00011-of-00034.safetensors with huggingface_hub
Verified signature388c280 -
Upload model-00010-of-00034.safetensors with huggingface_hub
Verified signaturec49001f -
Upload model-00009-of-00034.safetensors with huggingface_hub
Verified signatured83d297 -
Upload model-00008-of-00034.safetensors with huggingface_hub
Verified signature1aaeb36 -
Upload model-00007-of-00034.safetensors with huggingface_hub
Verified signature4fa61fb -
Upload model-00006-of-00034.safetensors with huggingface_hub
Verified signatured58ea28 -
Upload model-00005-of-00034.safetensors with huggingface_hub
Verified signature742fc13 -
Upload model-00004-of-00034.safetensors with huggingface_hub
Verified signature36f1660 -
Upload model-00003-of-00034.safetensors with huggingface_hub
Verified signature7a9b06d -
Upload model-00002-of-00034.safetensors with huggingface_hub
Verified signature958b9f8 -
Upload model-00001-of-00034.safetensors with huggingface_hub
Verified signaturec24a7bc -
Upload LICENSE with huggingface_hub
Verified signature91ef23b -
Upload generation_config.json with huggingface_hub
Verified signature3b403a0 -
Upload config.json with huggingface_hub
Verified signature053f2e7 -
Upload chat_template.jinja with huggingface_hub
Verified signature70d17cf