Instructions to use flytech/devchat-llama-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use flytech/devchat-llama-7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="flytech/devchat-llama-7b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("flytech/devchat-llama-7b")
model = AutoModelForCausalLM.from_pretrained("flytech/devchat-llama-7b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use flytech/devchat-llama-7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "flytech/devchat-llama-7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "flytech/devchat-llama-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/flytech/devchat-llama-7b

SGLang

How to use flytech/devchat-llama-7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "flytech/devchat-llama-7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "flytech/devchat-llama-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "flytech/devchat-llama-7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "flytech/devchat-llama-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use flytech/devchat-llama-7b with Docker Model Runner:
```
docker model run hf.co/flytech/devchat-llama-7b
```

devchat-llama-7b / last-checkpoint

Commit History

Training in progress, step 900, checkpoint

fe16d95

flytech commited on Sep 9, 2023

Training in progress, step 800, checkpoint

9d9b2ee

flytech commited on Sep 9, 2023

Training in progress, step 700, checkpoint

eddce0a

flytech commited on Sep 9, 2023

Training in progress, step 600, checkpoint

77ceeca

flytech commited on Sep 9, 2023

Training in progress, step 500, checkpoint

32628c0

flytech commited on Sep 9, 2023

Training in progress, step 400, checkpoint

e5cd618

flytech commited on Sep 9, 2023

Training in progress, step 300, checkpoint

498564f

flytech commited on Sep 9, 2023

Training in progress, step 200, checkpoint

7d11b45

flytech commited on Sep 9, 2023

Training in progress, step 100, checkpoint

229fcf2

flytech commited on Sep 9, 2023

Training in progress, step 4, checkpoint

94c64f2

flytech commited on Sep 9, 2023

Training in progress, step 2, checkpoint

f06f27b

flytech commited on Sep 9, 2023

Training in progress, step 200, checkpoint

fd03e8a

sirr commited on Sep 8, 2023

Training in progress, step 175, checkpoint

d53feaa

sirr commited on Sep 8, 2023

Training in progress, step 150, checkpoint

75c368f

sirr commited on Sep 8, 2023

Training in progress, step 125, checkpoint

fe59400

sirr commited on Sep 8, 2023

Training in progress, step 100, checkpoint

c1ac544

sirr commited on Sep 8, 2023

Training in progress, step 75, checkpoint

a101dab

sirr commited on Sep 8, 2023

Training in progress, step 50, checkpoint

c91b256

sirr commited on Sep 8, 2023

Training in progress, step 25, checkpoint

eeaf65a

sirr commited on Sep 8, 2023

Training in progress, step 100, checkpoint

270c901

sirr commited on Sep 8, 2023

Commit History

Training in progress, step 900, checkpoint fe16d95

Training in progress, step 800, checkpoint 9d9b2ee

Training in progress, step 700, checkpoint eddce0a

Training in progress, step 600, checkpoint 77ceeca

Training in progress, step 500, checkpoint 32628c0

Training in progress, step 400, checkpoint e5cd618

Training in progress, step 300, checkpoint 498564f

Training in progress, step 200, checkpoint 7d11b45

Training in progress, step 100, checkpoint 229fcf2

Training in progress, step 4, checkpoint 94c64f2

Training in progress, step 2, checkpoint f06f27b

Training in progress, step 200, checkpoint fd03e8a

Training in progress, step 175, checkpoint d53feaa

Training in progress, step 150, checkpoint 75c368f

Training in progress, step 125, checkpoint fe59400

Training in progress, step 100, checkpoint c1ac544

Training in progress, step 75, checkpoint a101dab

Training in progress, step 50, checkpoint c91b256

Training in progress, step 25, checkpoint eeaf65a

Training in progress, step 100, checkpoint 270c901

Training in progress, step 900, checkpoint

fe16d95

Training in progress, step 800, checkpoint

9d9b2ee

Training in progress, step 700, checkpoint

eddce0a

Training in progress, step 600, checkpoint

77ceeca

Training in progress, step 500, checkpoint

32628c0

Training in progress, step 400, checkpoint

e5cd618

Training in progress, step 300, checkpoint

498564f

Training in progress, step 200, checkpoint

7d11b45

Training in progress, step 100, checkpoint

229fcf2

Training in progress, step 4, checkpoint

94c64f2

Training in progress, step 2, checkpoint

f06f27b

Training in progress, step 200, checkpoint

fd03e8a

Training in progress, step 175, checkpoint

d53feaa

Training in progress, step 150, checkpoint

75c368f

Training in progress, step 125, checkpoint

fe59400

Training in progress, step 100, checkpoint

c1ac544

Training in progress, step 75, checkpoint

a101dab

Training in progress, step 50, checkpoint

c91b256

Training in progress, step 25, checkpoint

eeaf65a

Training in progress, step 100, checkpoint

270c901