Instructions to use openchat/openchat_3.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openchat/openchat_3.5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="openchat/openchat_3.5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openchat/openchat_3.5")
model = AutoModelForCausalLM.from_pretrained("openchat/openchat_3.5", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use openchat/openchat_3.5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openchat/openchat_3.5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openchat/openchat_3.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/openchat/openchat_3.5

SGLang

How to use openchat/openchat_3.5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openchat/openchat_3.5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openchat/openchat_3.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openchat/openchat_3.5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openchat/openchat_3.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use openchat/openchat_3.5 with Docker Model Runner:
```
docker model run hf.co/openchat/openchat_3.5
```

openchat_3.5

Commit History

Jun 14, 2025

15:36
add AIBOM⁠

sabato-nocera

Verified signature
61754a3

May 18, 2024

18:09
Update README.md

alpayariyak

Verified signature
0fc98e3

May 7, 2024

10:12
Update README.md

imone

Verified signature
c8ac815

Dec 24, 2023

06:41
Update README.md

SijieCheng

e4d87b9

Dec 13, 2023

08:18
Update README.md

alpayariyak

99d59d4

Dec 8, 2023

07:52
Update README.md

imone

2b2dbf0
07:12
Update README.md

imone

effe26a

Nov 23, 2023

07:38
Update README.md

imone

75a0c61

Nov 22, 2023

01:00
Update README.md

alpayariyak

7e65595

Nov 21, 2023

18:16
Add chat template (#27)⁠

imone

a11af6b

Nov 17, 2023

04:16
Create generation_config.json (#21)⁠

imone +1: fenglui

5b874a3

Nov 15, 2023

12:53
Update README.md

imone

600a344

Nov 9, 2023

05:31
Update model card metadata

imone

2d39b2e

Nov 8, 2023

16:30
Update dataset details in model card (#11)⁠

imone

37ca7ce

Nov 5, 2023

12:43
Update README.md

imone

0be788e
12:34
Update README.md

imone

b6c8863
11:08
Update README.md

imone

5563fa7
11:03
Update README.md

imone

17ccd19

Nov 4, 2023

12:24
Update README.md

imone

eebadee
12:09
Add baselines

imone

6814591

Nov 3, 2023

09:33
Update README.md

imone

4d04f08
09:29
Update README.md

imone

3618d9b
04:03
EOS should be 32000 (#4)⁠

imone +1: TheBloke

d4024c4
04:02
EOS should be 32000 (#3)⁠

imone +1: TheBloke

394df69

Nov 2, 2023

17:27
[doc] update README

tb-upce

f85ea3d
14:50
Update model card

tb-upce

17083ef

Nov 1, 2023

15:56
Add fast tokenizer support

imone

81689a2

Oct 30, 2023

13:36
bf16

imone

0ba9dfe
13:28
fp16

imone

1387ce5
06:03
weights

imone

1025f2b
05:59
initial commit

imone

f897c6b

Commit History

Jun 14, 2025

add AIBOM⁠

May 18, 2024

Update README.md

May 7, 2024

Update README.md

Dec 24, 2023

Update README.md

Dec 13, 2023

Update README.md

Dec 8, 2023

Update README.md

Update README.md

Nov 23, 2023

Update README.md

Nov 22, 2023

Update README.md

Nov 21, 2023

Add chat template (#27)⁠

Nov 17, 2023

Create generation_config.json (#21)⁠

Nov 15, 2023

Update README.md

Nov 9, 2023

Update model card metadata

Nov 8, 2023

Update dataset details in model card (#11)⁠

Nov 5, 2023

Update README.md

Update README.md

Update README.md

Update README.md

Nov 4, 2023

Update README.md

Add baselines

Nov 3, 2023

Update README.md

Update README.md

EOS should be 32000 (#4)⁠

EOS should be 32000 (#3)⁠

Nov 2, 2023

[doc] update README

Update model card

Nov 1, 2023

Add fast tokenizer support

Oct 30, 2023

bf16

fp16

weights

initial commit