Instructions to use RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16

SGLang

How to use RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16
```

DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16

Commit History

May 7

20:54
Fix chat_template crash when assistant message omits the `content` key⁠

qgallouedec HF Staff

Verified signature
0a852a0

Feb 27, 2025

08:11
Add reasoning evals

nm-research

Verified signature
9559a91

Feb 24, 2025

14:59
Update README.md

nm-research

Verified signature
ff42978

Feb 22, 2025

23:52
update tokenizer configs

nm-research

10b57d6

Feb 20, 2025

19:19
Update README.md

nm-research

Verified signature
cdcbf2a
15:56
Update README.md

nm-research

Verified signature
a9c8bbd
15:54
Update README.md

nm-research

Verified signature
4c56500

Feb 12, 2025

17:20
Update README.md

nm-research

Verified signature
37961c1
16:50
Create README.md

nm-research

Verified signature
e02bd08

Feb 4, 2025

17:57
Upload folder using huggingface_hub

nm-research

Verified signature
580efd3
17:55
initial commit

nm-research

Verified signature
4c0fa26

Commit History

May 7

Fix chat_template crash when assistant message omits the `content` key⁠

Feb 27, 2025

Add reasoning evals

Feb 24, 2025

Update README.md

Feb 22, 2025

update tokenizer configs

Feb 20, 2025

Update README.md

Update README.md

Update README.md

Feb 12, 2025

Update README.md

Create README.md

Feb 4, 2025

Upload folder using huggingface_hub

initial commit