Instructions to use bigscience/bloomz-7b1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bigscience/bloomz-7b1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bigscience/bloomz-7b1")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloomz-7b1")
model = AutoModelForCausalLM.from_pretrained("bigscience/bloomz-7b1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bigscience/bloomz-7b1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bigscience/bloomz-7b1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/bloomz-7b1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/bigscience/bloomz-7b1

SGLang

How to use bigscience/bloomz-7b1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bigscience/bloomz-7b1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/bloomz-7b1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bigscience/bloomz-7b1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/bloomz-7b1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use bigscience/bloomz-7b1 with Docker Model Runner:
```
docker model run hf.co/bigscience/bloomz-7b1
```

bloomz-7b1

Commit History

Feb 22, 2024

03:44
Adding `safetensors` variant of this model⁠

SFconvertbot

Verified signature
a53257f

May 27, 2023

17:25
Update README.md

Muennighoff

2f4c4f3

Feb 10, 2023

15:20
Fix architecture (#9)⁠

lewtun HF Staff

63d2053
14:33
Fix architecture (#8)⁠

cakiki +1: lewtun

55b1e40

Jan 19, 2023

10:05
Update tokenizer_config.json (#7)⁠

Muennighoff +1: ybelkada

229c446

Dec 3, 2022

12:38
Update README.md

Muennighoff

18a233a

Nov 18, 2022

17:31
Update README.md

Muennighoff

65f94ca

Nov 10, 2022

06:38
Remove eval

Muennighoff

0268979

Nov 4, 2022

00:22
Add paper

Muennighoff

f751571

Nov 3, 2022

23:58
Create README.md

Muennighoff

eb6a2c3

Oct 9, 2022

14:01
Logs

Muennighoff

ed75a42
12:09
Organize

Muennighoff

5754a9f
12:06
Rename

Muennighoff

d3cba1f

Sep 27, 2022

09:11
Add files

Muennighoff

899c702
09:00
initial commit

Muennighoff

f5692b9

Commit History

Feb 22, 2024

Adding `safetensors` variant of this model⁠

May 27, 2023

Update README.md

Feb 10, 2023

Fix architecture (#9)⁠

Fix architecture (#8)⁠

Jan 19, 2023

Update tokenizer_config.json (#7)⁠

Dec 3, 2022

Update README.md

Nov 18, 2022

Update README.md

Nov 10, 2022

Remove eval

Nov 4, 2022

Add paper

Nov 3, 2022

Create README.md

Oct 9, 2022

Logs

Organize

Rename

Sep 27, 2022

Add files

initial commit