Instructions to use LimYeri/CodeMind-Gemma-7B-QLoRA-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LimYeri/CodeMind-Gemma-7B-QLoRA-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")
model = AutoModelForCausalLM.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LimYeri/CodeMind-Gemma-7B-QLoRA-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LimYeri/CodeMind-Gemma-7B-QLoRA-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LimYeri/CodeMind-Gemma-7B-QLoRA-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LimYeri/CodeMind-Gemma-7B-QLoRA-4bit

SGLang

How to use LimYeri/CodeMind-Gemma-7B-QLoRA-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LimYeri/CodeMind-Gemma-7B-QLoRA-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LimYeri/CodeMind-Gemma-7B-QLoRA-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LimYeri/CodeMind-Gemma-7B-QLoRA-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LimYeri/CodeMind-Gemma-7B-QLoRA-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use LimYeri/CodeMind-Gemma-7B-QLoRA-4bit with Docker Model Runner:
```
docker model run hf.co/LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
```

LimYeri commited on Apr 18, 2024

Commit

afe10d4

verified ·

1 Parent(s): e2afb58

Update README.md

Browse files

Files changed (1) hide show

README.md +3 -2

README.md CHANGED Viewed

@@ -31,8 +31,9 @@ The model was fine-tuned using the following datasets:
 1. **LimYeri/LeetCode_with_Solutions**: This dataset contains Leetcode problems along with their hints, user solutions that have received at least 10 votes, and summaries of Leetcode solution videos from YouTube. These summaries have been processed using the Chain of Thought (CoT) method via commercial Large Language Model (LLM). The 'content' column houses the solutions and captions(CoT Summary), providing detailed explanations, thought processes, and step-by-step instructions for solving the coding problems.
 ## Training Procedure
-The model was fine-tuned using the Hugging Face Transformer library. The base model, [gemma-7b-it](https://huggingface.co/google/gemma-7b-it), was further trained on the combined dataset of LeetCode user solutions and YouTube video captions(CoT Summary).  This fine-tuning process was designed to enhance the model's understanding of coding concepts and problem-solving strategies, and improve its ability to generate relevant code snippets and explanations.
 ## Bias and Limitations
 - The model's knowledge is primarily based on the LeetCode user solutions and YouTube video captions(CoT Summary) used for fine-tuning. It may have limitations in handling coding problems or concepts that are not well-represented in the training data.
 - The model's responses are generated based on patterns and information learned from the training data. It may sometimes produce incorrect or suboptimal solutions. Users should always review and verify the generated code before using it in practice.

 1. **LimYeri/LeetCode_with_Solutions**: This dataset contains Leetcode problems along with their hints, user solutions that have received at least 10 votes, and summaries of Leetcode solution videos from YouTube. These summaries have been processed using the Chain of Thought (CoT) method via commercial Large Language Model (LLM). The 'content' column houses the solutions and captions(CoT Summary), providing detailed explanations, thought processes, and step-by-step instructions for solving the coding problems.
 ## Training Procedure
+- The model was fine-tuned using the Hugging Face Transformer library. The base model, [gemma-7b-it](https://huggingface.co/google/gemma-7b-it), was further trained on the combined dataset of LeetCode user solutions and YouTube video captions(CoT Summary).  This fine-tuning process was designed to enhance the model's understanding of coding concepts and problem-solving strategies, and improve its ability to generate relevant code snippets and explanations.
+- The model was trained using the QLoRA technique with 4-bit quantization on the dataset.
 ## Bias and Limitations
 - The model's knowledge is primarily based on the LeetCode user solutions and YouTube video captions(CoT Summary) used for fine-tuning. It may have limitations in handling coding problems or concepts that are not well-represented in the training data.
 - The model's responses are generated based on patterns and information learned from the training data. It may sometimes produce incorrect or suboptimal solutions. Users should always review and verify the generated code before using it in practice.