Instructions to use professorf/DeepSeek-R1-Distill-Llama-8B-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use professorf/DeepSeek-R1-Distill-Llama-8B-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16 # Run inference directly in the terminal: llama cli -hf professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16 # Run inference directly in the terminal: llama cli -hf professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16 # Run inference directly in the terminal: ./build/bin/llama cli -hf professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16
Use Docker
docker model run hf.co/professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16
- Llama new
- LM Studio
- Jan
- Ollama
How to use professorf/DeepSeek-R1-Distill-Llama-8B-gguf with Ollama:
ollama run hf.co/professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16
- Unsloth Desktop
- Docker Model Runner
How to use professorf/DeepSeek-R1-Distill-Llama-8B-gguf with Docker Model Runner:
docker model run hf.co/professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16
- Lemonade
How to use professorf/DeepSeek-R1-Distill-Llama-8B-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull professorf/DeepSeek-R1-Distill-Llama-8B-gguf:F16
Run and chat with the model
lemonade run user.DeepSeek-R1-Distill-Llama-8B-gguf-F16
List all available models
lemonade list
- Atomic Chat
by Professor Nick V. Flor
For research reproducibility purposes
[🏠Homepage] | [🤖 Chat with DeepSeek LLM] | [Discord] | [Wechat(微信)]
1. Introduction of Deepseek LLM
Introducing DeepSeek LLM, an advanced language model comprising 7 billion parameters. It has been trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese. In order to foster research, we have made DeepSeek LLM 7B/67B Base and DeepSeek LLM 7B/67B Chat open source for the research community.
2. Model Summary
deepseek-llm-7b-chat is a 7B parameter model initialized from deepseek-llm-7b-base and fine-tuned on extra instruction data.
- Home Page: DeepSeek
- Repository: deepseek-ai/deepseek-LLM
- Chat With DeepSeek LLM: DeepSeek-LLM
3. How to Use
Here give some examples of how to use our model.
Chat Completion
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
model_name = "deepseek-ai/deepseek-llm-7b-chat"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
model.generation_config = GenerationConfig.from_pretrained(model_name)
model.generation_config.pad_token_id = model.generation_config.eos_token_id
messages = [
{"role": "user", "content": "Who are you?"}
]
input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
print(result)
Avoiding the use of the provided function apply_chat_template, you can also interact with our model following the sample template. Note that messages should be replaced by your input.
User: {messages[0]['content']}
Assistant: {messages[1]['content']}<|end▁of▁sentence|>User: {messages[2]['content']}
Assistant:
Note: By default (add_special_tokens=True), our tokenizer automatically adds a bos_token (<|begin▁of▁sentence|>) before the input text. Additionally, since the system prompt is not compatible with this version of our models, we DO NOT RECOMMEND including the system prompt in your input.
4. License
This code repository is licensed under the MIT License. The use of DeepSeek LLM models is subject to the Model License. DeepSeek LLM supports commercial use.
See the LICENSE-MODEL for more details.
5. Contact
If you have any questions, please raise an issue or contact us at service@deepseek.com.
- Downloads last month
- 7
8-bit
16-bit