Instructions to use nvidia/Minitron-8B-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/Minitron-8B-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nvidia/Minitron-8B-Base")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nvidia/Minitron-8B-Base")
model = AutoModelForCausalLM.from_pretrained("nvidia/Minitron-8B-Base", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/Minitron-8B-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/Minitron-8B-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Minitron-8B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/nvidia/Minitron-8B-Base

SGLang

How to use nvidia/Minitron-8B-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/Minitron-8B-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Minitron-8B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/Minitron-8B-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Minitron-8B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use nvidia/Minitron-8B-Base with Docker Model Runner:
```
docker model run hf.co/nvidia/Minitron-8B-Base
```

Minitron-8B-Base

Commit History

Aug 15, 2024

08:19
Adding `safetensors` variant of this model⁠

SFconvertbot

Verified signature
e96bd77

Aug 14, 2024

00:29
Add copyright notice

srvm

34eef63

Aug 13, 2024

21:12
Typo

srvm

54057c9
21:11
Update model name

srvm

6d40552

Jul 23, 2024

17:36
Add evaluation preview

srvm

019a9d0
07:24
Clarify commit ID

srvm

66505a7
00:24
Add arXiv details

srvm

7bc9a51

Jul 22, 2024

20:53
Point to correct commit

srvm

615c28e
17:29
Add HF checkpoint and metadata

srvm

06da12d
03:13
Add nemo checkpoint

srvm

3eeabf8
03:08
Update gitattributes

srvm

794fdba
01:52
Add requirements, update gitattributes

srvm

be3c438
01:52
Add initial README

srvm

44cbfe3
00:15
Add LICENSE

srvm

3bda58b

Jul 19, 2024

20:12
initial commit

Pavlo

Verified signature
82aefb2

Commit History

Aug 15, 2024

Adding `safetensors` variant of this model⁠

Aug 14, 2024

Add copyright notice

Aug 13, 2024

Typo

Update model name

Jul 23, 2024

Add evaluation preview

Clarify commit ID

Add arXiv details

Jul 22, 2024

Point to correct commit

Add HF checkpoint and metadata

Add nemo checkpoint

Update gitattributes

Add requirements, update gitattributes

Add initial README

Add LICENSE

Jul 19, 2024

initial commit