Instructions to use trl-internal-testing/tiny-random-LlamaForCausalLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use trl-internal-testing/tiny-random-LlamaForCausalLM with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="trl-internal-testing/tiny-random-LlamaForCausalLM")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("trl-internal-testing/tiny-random-LlamaForCausalLM")
model = AutoModelForCausalLM.from_pretrained("trl-internal-testing/tiny-random-LlamaForCausalLM", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use trl-internal-testing/tiny-random-LlamaForCausalLM with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "trl-internal-testing/tiny-random-LlamaForCausalLM"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trl-internal-testing/tiny-random-LlamaForCausalLM",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/trl-internal-testing/tiny-random-LlamaForCausalLM

SGLang

How to use trl-internal-testing/tiny-random-LlamaForCausalLM with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "trl-internal-testing/tiny-random-LlamaForCausalLM" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trl-internal-testing/tiny-random-LlamaForCausalLM",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "trl-internal-testing/tiny-random-LlamaForCausalLM" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trl-internal-testing/tiny-random-LlamaForCausalLM",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use trl-internal-testing/tiny-random-LlamaForCausalLM with Docker Model Runner:
```
docker model run hf.co/trl-internal-testing/tiny-random-LlamaForCausalLM
```

tiny-random-LlamaForCausalLM

Commit History

Upload tokenizer

2c542e4
verified

ybelkada HF Staff commited on Apr 23, 2024

Update generation_config.json

e47f42c
verified

ybelkada HF Staff commited on Apr 23, 2024

Adding `safetensors` variant of this model (#1)

a2230a9
verified

ybelkada HF Staff

SFconvertbot commited on Apr 10, 2024

Upload tokenizer

a3361c1

ybelkada HF Staff commited on Mar 29, 2023

Upload LlamaForCausalLM

5dad3ab

ybelkada HF Staff commited on Mar 29, 2023

initial commit

d33b1ec

Younes Belkada commited on Mar 29, 2023

Commit History

Upload tokenizer 2c542e4 verified

Update generation_config.json e47f42c verified

Adding `safetensors` variant of this model (#1) a2230a9 verified

Upload tokenizer a3361c1

Upload LlamaForCausalLM 5dad3ab