Instructions to use bigscience/mt0-xxl-mt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bigscience/mt0-xxl-mt with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bigscience/mt0-xxl-mt")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("bigscience/mt0-xxl-mt")
model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/mt0-xxl-mt", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bigscience/mt0-xxl-mt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bigscience/mt0-xxl-mt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/mt0-xxl-mt",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/bigscience/mt0-xxl-mt

SGLang

How to use bigscience/mt0-xxl-mt with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bigscience/mt0-xxl-mt" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/mt0-xxl-mt",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bigscience/mt0-xxl-mt" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/mt0-xxl-mt",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use bigscience/mt0-xxl-mt with Docker Model Runner:
```
docker model run hf.co/bigscience/mt0-xxl-mt
```

mt0-xxl-mt

Commit History

Jun 6, 2024

20:20
Adding `safetensors` variant of this model⁠

SFconvertbot

Verified signature
879b255

Jul 25, 2023

11:13
Update README.md

Muennighoff

37ebb4f

Mar 8, 2023

09:52
Update README.md (#2)⁠

TimeRobber

5fd24a4

Dec 3, 2022

12:34
Correct pos

Muennighoff

8d86319

Nov 18, 2022

17:36
Update README.md

Muennighoff

07efa51

Nov 5, 2022

14:34
Update README.md

Muennighoff

ce4ad61
12:51
Update inference

Muennighoff

11d28f3

Nov 4, 2022

00:50
Update README.md

Muennighoff

8aa74f1

Nov 1, 2022

13:34
Merge branch 'checkpoint_1007000' into main

thomasw21

3b742ce

Oct 29, 2022

21:17
Update README.md

TimeRobber

d6061c1

Oct 28, 2022

21:45
Update README.md

TimeRobber

488b707
21:43
Update README.md

TimeRobber

b954a57
21:33
WIP: README.md (#1)⁠

TimeRobber

6f9f453
15:24
Add checkpoint_1007000

thomasw21

9ec8027

Oct 27, 2022

21:21
initial commit

Thomas Wang

739c994

Commit History

Jun 6, 2024

Adding `safetensors` variant of this model⁠

Jul 25, 2023

Update README.md

Mar 8, 2023

Update README.md (#2)⁠

Dec 3, 2022

Correct pos

Nov 18, 2022

Update README.md

Nov 5, 2022

Update README.md

Update inference

Nov 4, 2022

Update README.md

Nov 1, 2022

Merge branch 'checkpoint_1007000' into main

Oct 29, 2022

Update README.md

Oct 28, 2022

Update README.md

Update README.md

WIP: README.md (#1)⁠

Add checkpoint_1007000

Oct 27, 2022

initial commit