Instructions to use openbmb/MiniCPM3-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openbmb/MiniCPM3-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="openbmb/MiniCPM3-4B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openbmb/MiniCPM3-4B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("openbmb/MiniCPM3-4B", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use openbmb/MiniCPM3-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openbmb/MiniCPM3-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM3-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/openbmb/MiniCPM3-4B

SGLang

How to use openbmb/MiniCPM3-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openbmb/MiniCPM3-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM3-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openbmb/MiniCPM3-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM3-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use openbmb/MiniCPM3-4B with Docker Model Runner:
```
docker model run hf.co/openbmb/MiniCPM3-4B
```

MiniCPM3-4B

Commit History

Nov 14, 2024

13:33
Adding `safetensors` variant of this model⁠

SFconvertbot

Verified signature
7bbf111
07:59
fix: set add_bos_token=true

FrankC0st1e

f3e7d16

Oct 28, 2024

07:26
remove minicpm tokenizer (#29)⁠

cppowboy

Verified signature
e8a65f6
02:26
Fix function calling parameter string quoting (#24)⁠

cppowboy +1: CISCai

Verified signature
bfe2f6c

Oct 8, 2024

06:06
Add full tool calling support to chat template (#20)⁠

cppowboy +1: CISCai

Verified signature
c3c3550

Sep 27, 2024

06:16
Update tokenization_minicpm.py

cppowboy

Verified signature
7f8e63d

Sep 18, 2024

13:00
Update tokenization_minicpm.py

cppowboy

Verified signature
a9d3817
12:40
Update tokenization_minicpm.py

cppowboy

Verified signature
508a16b

Sep 12, 2024

13:15
Add `model_type` to config (#17)⁠

FrankC0st1e +1: pi314ever-intel

Verified signature
6fcf8b4

Sep 7, 2024

03:38
add library tag (#8)⁠

neoz +1: davanstrien

Verified signature
75f9f10

Sep 6, 2024

13:22
Update README.md

neoz

Verified signature
8d724f7

Sep 5, 2024

16:54
Update tokenization_minicpm.py

FrankC0st1e

Verified signature
f5d31d8
15:54
Update README.md

neoz

Verified signature
e618f3a
12:28
fix loss in modeling_minicpm.py

FrankC0st1e

fc345c8
11:15
update bfcl and livecodebench version

FrankC0st1e

65d6b09
10:26
Update README.md

FrankC0st1e

Verified signature
52fc630
10:24
Update README.md

FrankC0st1e

Verified signature
a18cc45
09:45
Update README.md

FrankC0st1e

Verified signature
84b4815
08:27
Update README.md

FrankC0st1e

Verified signature
f03dca5
08:22
Update README.md

FrankC0st1e

Verified signature
d7cca6a
08:05
update readme

FrankC0st1e

65aff61
07:58
add vllm inference example

FrankC0st1e

49e4794
06:02
update readme

FrankC0st1e

6c769fa
03:44
fix: empty system prompt

FrankC0st1e

43b498b

Sep 4, 2024

14:39
fix bug in .py

FrankC0st1e

a4f2dcb
13:33
change name to minicpm3

FrankC0st1e

8fa0de6

Sep 3, 2024

13:16
fix name_or_path in config.json

FrankC0st1e

91a5e0a
13:03
initial commit

yanhui

4a38946
12:47
initial commit

FrankC0st1e

Verified signature
760549e

Commit History

Nov 14, 2024

Adding `safetensors` variant of this model⁠

fix: set add_bos_token=true

Oct 28, 2024

remove minicpm tokenizer (#29)⁠

Fix function calling parameter string quoting (#24)⁠

Oct 8, 2024

Add full tool calling support to chat template (#20)⁠

Sep 27, 2024

Update tokenization_minicpm.py

Sep 18, 2024

Update tokenization_minicpm.py

Update tokenization_minicpm.py

Sep 12, 2024

Add `model_type` to config (#17)⁠

Sep 7, 2024

add library tag (#8)⁠

Sep 6, 2024

Update README.md

Sep 5, 2024

Update tokenization_minicpm.py

Update README.md

fix loss in modeling_minicpm.py

update bfcl and livecodebench version

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

update readme

add vllm inference example

update readme

fix: empty system prompt

Sep 4, 2024

fix bug in .py

change name to minicpm3

Sep 3, 2024

fix name_or_path in config.json

initial commit

initial commit