Instructions to use TheBloke/CodeLlama-13B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/CodeLlama-13B-Instruct-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/CodeLlama-13B-Instruct-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("TheBloke/CodeLlama-13B-Instruct-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use TheBloke/CodeLlama-13B-Instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use TheBloke/CodeLlama-13B-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/CodeLlama-13B-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-13B-Instruct-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M

SGLang

How to use TheBloke/CodeLlama-13B-Instruct-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/CodeLlama-13B-Instruct-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-13B-Instruct-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/CodeLlama-13B-Instruct-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-13B-Instruct-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use TheBloke/CodeLlama-13B-Instruct-GGUF with Ollama:
```
ollama run hf.co/TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use TheBloke/CodeLlama-13B-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M
```

Lemonade

How to use TheBloke/CodeLlama-13B-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheBloke/CodeLlama-13B-Instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.CodeLlama-13B-Instruct-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

CodeLlama-13B-Instruct-GGUF

Commit History

Sep 27, 2023

12:46
Update base_model formatting

TheBloke

82f1dd9

Sep 19, 2023

00:31
Upload README.md

TheBloke

61ba268

Sep 7, 2023

11:12
Upload README.md

TheBloke

fdfeb98

Sep 5, 2023

14:12
Upload README.md

TheBloke

34d9824
00:14
Upload README.md

TheBloke

7a6a09b
00:14
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

317deae
00:14
Add Llama 2 license files

TheBloke

d4b2385
00:14
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

e94db8d
00:13
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

b52bcee
00:12
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

295f6e2
00:11
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

c67fe9f
00:10
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

449e1b8
00:09
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

aa77850
00:08
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

3f3de95
00:07
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

1b4ad9c
00:06
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

15a3e70
00:05
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

52408ce
00:04
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

749b436
00:04
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

5a44264

Aug 26, 2023

15:45
Initial GGUF model commit

TheBloke

a738f7e
15:45
Initial GGUF model commit

TheBloke

421a736
15:45
Add Llama 2 license files

TheBloke

4d41d05
15:45
Add Llama 2 license files

TheBloke

f365ce1
15:45
Add Llama 2 license files

TheBloke

83f9dc7
15:43
Initial GGUF model commit

TheBloke

06636dc
15:43
Initial GGUF model commit

TheBloke

b38e0bd
15:43
Add Llama 2 license files

TheBloke

53518f1
15:43
Add Llama 2 license files

TheBloke

b99b85c
15:43
Add Llama 2 license files

TheBloke

f51f2f7
11:14
Initial GGUF model commit

TheBloke

9381883
11:14
Initial GGUF model commit

TheBloke

49658a9
11:14
Add Llama 2 license files

TheBloke

8bf20bc
11:14
Add Llama 2 license files

TheBloke

83aa9d0
11:14
Add Llama 2 license files

TheBloke

40a69c1

Aug 25, 2023

15:33
Initial GGUF model commit

TheBloke

7c41abe
15:33
Initial GGUF model commit

TheBloke

3071a20
15:33
Add Llama 2 license files

TheBloke

474929e
15:33
Add Llama 2 license files

TheBloke

f14ad5f
15:33
Add Llama 2 license files

TheBloke

efe807d
15:31
Initial GGUF model commit

TheBloke

84a6ced
15:31
Initial GGUF model commit

TheBloke

df23a3b
15:31
Add Llama 2 license files

TheBloke

86059cd
15:31
Add Llama 2 license files

TheBloke

3ad0d03
15:31
Add Llama 2 license files

TheBloke

d6f9215
08:26
Update README.md

TheBloke

0f16170

Aug 24, 2023

20:36
Initial GGUF model commit

TheBloke

00e3a4e
20:36
Initial GGUF model commit

TheBloke

c74af85
20:36
Add Llama 2 license files

TheBloke

4edbe3c
20:36
Add Llama 2 license files

TheBloke

1788fe6
20:36
Add Llama 2 license files

TheBloke

750501c

Commit History

Sep 27, 2023

Update base_model formatting

Sep 19, 2023

Upload README.md

Sep 7, 2023

Upload README.md

Sep 5, 2023

Upload README.md

Upload README.md

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Add Llama 2 license files

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Aug 26, 2023

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Aug 25, 2023

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Update README.md

Aug 24, 2023

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files