Instructions to use tencent/Hy3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tencent/Hy3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tencent/Hy3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tencent/Hy3")
model = AutoModelForCausalLM.from_pretrained("tencent/Hy3", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tencent/Hy3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tencent/Hy3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hy3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tencent/Hy3

SGLang

How to use tencent/Hy3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tencent/Hy3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hy3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tencent/Hy3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hy3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tencent/Hy3 with Docker Model Runner:
```
docker model run hf.co/tencent/Hy3
```

Thank you

#23

by mileshighclub - opened 11 days ago

Discussion

mileshighclub

11 days ago

•

edited 11 days ago

I want to thank the team for this outstanding work.

I am running the full-weight BF16 at home (8U server with 8 cards and 768gb memory total - my electricity bill is awesome and getting even better because I am going to add second one). Having full Hy3 on local hardware has been a great experience, and it has become my go-to model.

I'm even able to easily fit an 8 bit quant on 4 cards while retaining strong performance, which for me means having a capable model with headroom to run other inferencing workloads in tandem, extending the creative liberty I can take with my personal AI projects.

I believe in having complete custody over my data, and I don't like to build castles on shifting sand. While I live in the United States, US frontier models have overstepped my hard boundaries. They now have way too many paranoid measures to where they have become insulting to use, and those recent changes diminish their potential for productive work as well as my desire to use them. It is clear that our leading AI companies are currently just as paranoid about their own domestic customers as they are about Chinese AI models.

Hy3 does not need to be the latest Anthropic model to be more appealing to me, because it respects me as the user in terms of digital freedom, and it does not preemptively accuse me of wrongdoing for asking it to work on legitimate items such as a custom GPU kernel. I'm happy to bear with its limitations and creatively work around them, because Hy3 works with me and has never felt humiliating to use.

After 12 years in consulting (data automation related to corporate finance), I plan to start my own business soon which involves bespoke automation workflows, and part of that will be to utilize best-in-class Chinese models whenever they would be a best-fit tool in a process. I will try reaching out to Tencent at that time (and won't be surprised if I'm too small of a player, but it's worth a shot). For now, I've started adding Tencent shares to my portfolio, because I am confident that there will be more great things to follow from your company and team.

Thanks again for this significant contribution to the open-weight community.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment