Instructions to use RedHatAI/granite-3.1-2b-instruct-quantized.w8a8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/granite-3.1-2b-instruct-quantized.w8a8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/granite-3.1-2b-instruct-quantized.w8a8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/granite-3.1-2b-instruct-quantized.w8a8")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/granite-3.1-2b-instruct-quantized.w8a8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/granite-3.1-2b-instruct-quantized.w8a8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/granite-3.1-2b-instruct-quantized.w8a8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-instruct-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/granite-3.1-2b-instruct-quantized.w8a8

SGLang

How to use RedHatAI/granite-3.1-2b-instruct-quantized.w8a8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/granite-3.1-2b-instruct-quantized.w8a8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-instruct-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/granite-3.1-2b-instruct-quantized.w8a8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-instruct-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/granite-3.1-2b-instruct-quantized.w8a8 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/granite-3.1-2b-instruct-quantized.w8a8
```

granite-3.1-2b-instruct-quantized.w8a8

Commit History

Update README.md

ba87c9a
verified

shubhrapandit commited on Feb 28, 2025

Update README.md

9b4266a
verified

nm-research commited on Jan 25, 2025

Update README.md

7a7c635
verified

nm-research commited on Jan 24, 2025

Update README.md

f781824
verified

nm-research commited on Jan 20, 2025

Update README.md

debee57
verified

nm-research commited on Jan 16, 2025

Update README.md

613a7f4
verified

nm-research commited on Jan 16, 2025

Update README.md

0d328c8
verified

nm-research commited on Jan 16, 2025

Update README.md

ba744f2
verified

nm-research commited on Jan 10, 2025

Create README.md

c4d11f6
verified

nm-research commited on Jan 10, 2025

Add model files

2af28b5

Shubhra Pandit commited on Jan 10, 2025

initial commit

7164a9e
verified

nm-research commited on Jan 7, 2025

Commit History

Update README.md ba87c9a verified

Update README.md 9b4266a verified

Update README.md 7a7c635 verified

Update README.md f781824 verified

Update README.md debee57 verified

Update README.md 613a7f4 verified

Update README.md 0d328c8 verified

Update README.md ba744f2 verified

Create README.md c4d11f6 verified

Add model files 2af28b5

initial commit 7164a9e verified

Update README.md

ba87c9a
verified

Update README.md

9b4266a
verified

Update README.md

7a7c635
verified

Update README.md

f781824
verified

Update README.md

debee57
verified

Update README.md

613a7f4
verified

Update README.md

0d328c8
verified

Update README.md

ba744f2
verified

Create README.md

c4d11f6
verified

Add model files

2af28b5

initial commit

7164a9e
verified