Instructions to use bigcode/starcoder2-15b-instruct-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bigcode/starcoder2-15b-instruct-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bigcode/starcoder2-15b-instruct-v0.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("bigcode/starcoder2-15b-instruct-v0.1")
model = AutoModelForCausalLM.from_pretrained("bigcode/starcoder2-15b-instruct-v0.1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bigcode/starcoder2-15b-instruct-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bigcode/starcoder2-15b-instruct-v0.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigcode/starcoder2-15b-instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bigcode/starcoder2-15b-instruct-v0.1

SGLang

How to use bigcode/starcoder2-15b-instruct-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bigcode/starcoder2-15b-instruct-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigcode/starcoder2-15b-instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bigcode/starcoder2-15b-instruct-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigcode/starcoder2-15b-instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bigcode/starcoder2-15b-instruct-v0.1 with Docker Model Runner:
```
docker model run hf.co/bigcode/starcoder2-15b-instruct-v0.1
```

yuxiang630 commited on Apr 26, 2024

Commit

cee4ec0

1 Parent(s): ee5afd4

Add evaluation and improve wording

Browse files

Files changed (3) hide show

README.md +9 -3
assets/sc2-instruct-evalplus.png +0 -0
assets/sc2-instruct-lcb-ds.png +0 -0

README.md CHANGED Viewed

@@ -104,7 +104,7 @@ We introduce StarCoder2-15B-Instruct-v0.1, the very first entirely self-aligned
 ### Intended use
-The model is exclusively trained with 50k **singe-turn** instruction-response pairs focused on **code generation**. It is supposed to answer **implementation-related instructions**. Instructions in other formats may result in unexpected outcomes. In such cases, we recommend providing a **response prefix** or a **one-shot example** to guide the model.
 Here is an example to get started with the model using the [transformers](https://huggingface.co/docs/transformers/index) library:
@@ -169,9 +169,15 @@ def sum_list_of_integers(numbers):
 ### Bias, Risks, and Limitations
-StarCoder2-15B-Instruct-v0.1 is primarily finetuned for Python code generation tasks that can be verified through execution, which may lead to biases and limitations. For example, the model may not accurately follow instructions that constrains the output format and may have limitations in its performance with other programming languages and out-of-domain coding problems. In such cases, we recommend providing a response prefix or a one-shot example to guide the model.
-StarCoder2-15B-Instruct-v0.1 also inherits the bias, risks, and limitations from its base StarCoder2-15B model. For more information, please refer to the [StarCoder2-15B model card](https://huggingface.co/bigcode/starcoder2-15b).
 ## Training Details

 ### Intended use
+The model is designed to respond to **coding-related instructions** in a single turn. Instructions in other styles may result in less accurate responses
 Here is an example to get started with the model using the [transformers](https://huggingface.co/docs/transformers/index) library:
 ### Bias, Risks, and Limitations
+StarCoder2-15B-Instruct-v0.1 is primarily finetuned for Python code generation tasks that can be verified through execution, which may lead to certain biases and limitations. For example, the model might not adhere strictly to instructions that dictate the output format. In these situations, it's beneficial to provide a **response prefix** or a **one-shot example** to steer the model’s output. Additionally, the model may have limitations with other programming languages and out-of-domain coding tasks.
+The model also inherits the bias, risks, and limitations from its base StarCoder2-15B model. For more information, please refer to the [StarCoder2-15B model card](https://huggingface.co/bigcode/starcoder2-15b).
+## Evaluation on EvalPlus, LiveCodeBench, and DS-1000
+![EvalPlus](assets/sc2-instruct-evalplus.png)
+![LiveCodeBench and DS-1000](assets/sc2-instruct-lcb-ds.png)
 ## Training Details

assets/sc2-instruct-evalplus.png ADDED Viewed

assets/sc2-instruct-lcb-ds.png ADDED Viewed