Instructions to use tiny-random/deepseek-v3.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiny-random/deepseek-v3.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiny-random/deepseek-v3.1", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiny-random/deepseek-v3.1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("tiny-random/deepseek-v3.1", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiny-random/deepseek-v3.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiny-random/deepseek-v3.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/deepseek-v3.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiny-random/deepseek-v3.1

SGLang

How to use tiny-random/deepseek-v3.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiny-random/deepseek-v3.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/deepseek-v3.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiny-random/deepseek-v3.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/deepseek-v3.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiny-random/deepseek-v3.1 with Docker Model Runner:
```
docker model run hf.co/tiny-random/deepseek-v3.1
```

deepseek-v3.1 / README.md

yujiepan

Upload folder using huggingface_hub

0cf9bd4 verified 12 months ago

preview code

Raw

History Blame Contribute Delete

10.4 kB

	---
	library_name: transformers
	pipeline_tag: text-generation
	inference: true
	widget:
	- text: Hello!
	example_title: Hello world
	group: Python
	base_model:
	- deepseek-ai/DeepSeek-V3.1
	---

	This tiny model is for debugging. It is randomly initialized with the config adapted from [deepseek-ai/DeepSeek-V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1).

	### Example usage:

	- vLLM

	```bash
	python -m vllm.entrypoints.openai.api_server \
	--tensor-parallel-size 2 \
	--model tiny-random/deepseek-v3.1 \
	--trust-remote-code \
	--speculative-config='{"method": "deepseek_mtp", "num_speculative_tokens": 1}'
	```

	- Transformers

	```python
	import torch
	import transformers

	model_id = "tiny-random/deepseek-v3.1"
	pipe = transformers.pipelines.pipeline(
	'text-generation',
	model=model_id,
	trust_remote_code=True,
	device_map='cuda',
	torch_dtype=torch.bfloat16,
	)
	r = pipe.model(torch.tensor([[1, 2, 3]], dtype=torch.int64).cuda(), attention_mask=None, use_cache=False)
	print(r)
	```

	### Codes to create this repo:

	```python
	import json
	from copy import deepcopy
	from pathlib import Path

	import accelerate
	import torch
	import torch.nn as nn
	from huggingface_hub import file_exists, hf_hub_download
	from transformers import (
	AutoConfig,
	AutoModelForCausalLM,
	AutoProcessor,
	AutoTokenizer,
	GenerationConfig,
	set_seed,
	)
	from transformers.models.glm4_moe.modeling_glm4_moe import Glm4MoeRMSNorm
	source_model_id = "deepseek-ai/DeepSeek-V3.1"
	save_folder = "/tmp/tiny-random/deepseek-v3.1"

	Path(save_folder).mkdir(parents=True, exist_ok=True)
	tokenizer = AutoTokenizer.from_pretrained(source_model_id, trust_remote_code=True)
	tokenizer.save_pretrained(save_folder)

	with open(hf_hub_download(source_model_id, filename='config.json', repo_type='model', cache_dir='/tmp/'), 'r', encoding='utf-8') as f:
	config_json = json.load(f)
	for k, v in config_json['auto_map'].items():
	config_json['auto_map'][k] = f'{source_model_id}--{v}'
	config_json.update({
	'first_k_dense_replace': 1,
	'num_hidden_layers': 2,
	'hidden_size': 8,
	'intermediate_size': 64,
	'kv_lora_rank': 384,
	'moe_intermediate_size': 64,
	'n_routed_experts': 32,
	'n_shared_experts': 1,
	'num_attention_heads': 4,
	'num_experts_per_tok': 8,
	'num_key_value_heads': 4,
	'q_lora_rank': 32,
	'qk_nope_head_dim': 64,
	'qk_rope_head_dim': 192, # vllm mla kernel supports 576 only, FA supports head dim <= 256
	'v_head_dim': 64,
	'tie_word_embeddings': False,
	})
	del config_json['quantization_config']
	with open(f"{save_folder}/config.json", "w", encoding='utf-8') as f:
	json.dump(config_json, f, indent=2)

	config = AutoConfig.from_pretrained(
	save_folder,
	trust_remote_code=True,
	)
	print(config)
	torch.set_default_dtype(torch.bfloat16)
	model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)

	if file_exists(filename="generation_config.json", repo_id=source_model_id, repo_type='model'):
	model.generation_config = GenerationConfig.from_pretrained(
	source_model_id, trust_remote_code=True,
	)

	class SharedHead(nn.Module):
	def __init__(self, config) -> None:
	super().__init__()
	from transformers.models.glm4_moe.modeling_glm4_moe import Glm4MoeRMSNorm
	self.norm = Glm4MoeRMSNorm(config.hidden_size, eps=config.rms_norm_eps)

	last_extra_layer = model.model.layers[0].__class__(config, layer_idx=config.num_hidden_layers)
	last_extra_layer.enorm = Glm4MoeRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
	last_extra_layer.hnorm = Glm4MoeRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
	last_extra_layer.eh_proj = nn.Linear(config.hidden_size * 2, config.hidden_size, bias=False)
	last_extra_layer.shared_head = SharedHead(config=config)
	model.model.layers.append(last_extra_layer)

	torch.set_default_dtype(torch.float32)
	set_seed(42)
	model = model.cpu() # cpu is more stable for random initialization across machines
	with torch.no_grad():
	for name, p in sorted(model.named_parameters()):
	torch.nn.init.normal_(p, 0, 0.1)
	print(name, p.shape)

	last_extra_layer.shared_head.head = deepcopy(model.get_output_embeddings())
	last_extra_layer.embed_tokens = deepcopy(model.get_input_embeddings())
	model.save_pretrained(save_folder)
	with open(f"{save_folder}/config.json", "r", encoding='utf-8') as f:
	config_json = json.load(f)
	config_json['auto_map'] = {k: v.split('--')[-1] for k, v in config_json['auto_map'].items()}
	with open(f"{save_folder}/config.json", "w", encoding='utf-8') as f:
	json.dump(config_json, f, indent=2)
	hf_hub_download(source_model_id, filename='modeling_deepseek.py', repo_type='model',
	local_dir=save_folder, local_dir_use_symlinks=False, cache_dir='/tmp/')
	with open(f'{save_folder}/modeling_deepseek.py', 'r', encoding='utf-8') as f:
	codes = f.read()
	codes = codes.replace(
	"past_length = past_key_values.seen_tokens",
	"past_length = past_key_values.seen_tokens if hasattr(past_key_values, 'seen_tokens') else past_key_values.get_seq_length() # fix cache api deprecation"
	)
	codes = codes.replace(
	"max_cache_length = past_key_values.get_max_length()",
	"max_cache_length = past_key_values.get_max_length() if hasattr(past_key_values, 'get_max_length') else past_key_values.get_max_cache_shape() # fix cache api deprecation"
	)
	codes = codes.replace(
	"past_key_value.get_usable_length(",
	"getattr(past_key_value, 'get_usable_length', lambda args, *kwargs: past_key_value.get_seq_length())("
	)
	codes = codes.replace(
	"past_key_values.get_usable_length(",
	"getattr(past_key_values, 'get_usable_length', lambda args, *kwargs: past_key_values.get_seq_length())("
	)
	with open(f'{save_folder}/modeling_deepseek.py', 'w', encoding='utf-8') as f:
	f.write(codes)
	```

	### Printing the model:

	```text
	DeepseekV3ForCausalLM(
	(model): DeepseekV3Model(
	(embed_tokens): Embedding(129280, 8)
	(layers): ModuleList(
	(0): DeepseekV3DecoderLayer(
	(self_attn): DeepseekV3Attention(
	(q_a_proj): Linear(in_features=8, out_features=32, bias=False)
	(q_a_layernorm): DeepseekV3RMSNorm()
	(q_b_proj): Linear(in_features=32, out_features=1024, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=8, out_features=576, bias=False)
	(kv_a_layernorm): DeepseekV3RMSNorm()
	(kv_b_proj): Linear(in_features=384, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=8, bias=False)
	(rotary_emb): DeepseekV3YarnRotaryEmbedding()
	)
	(mlp): DeepseekV3MLP(
	(gate_proj): Linear(in_features=8, out_features=64, bias=False)
	(up_proj): Linear(in_features=8, out_features=64, bias=False)
	(down_proj): Linear(in_features=64, out_features=8, bias=False)
	(act_fn): SiLU()
	)
	(input_layernorm): DeepseekV3RMSNorm()
	(post_attention_layernorm): DeepseekV3RMSNorm()
	)
	(1): DeepseekV3DecoderLayer(
	(self_attn): DeepseekV3Attention(
	(q_a_proj): Linear(in_features=8, out_features=32, bias=False)
	(q_a_layernorm): DeepseekV3RMSNorm()
	(q_b_proj): Linear(in_features=32, out_features=1024, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=8, out_features=576, bias=False)
	(kv_a_layernorm): DeepseekV3RMSNorm()
	(kv_b_proj): Linear(in_features=384, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=8, bias=False)
	(rotary_emb): DeepseekV3YarnRotaryEmbedding()
	)
	(mlp): DeepseekV3MoE(
	(experts): ModuleList(
	(0-31): 32 x DeepseekV3MLP(
	(gate_proj): Linear(in_features=8, out_features=64, bias=False)
	(up_proj): Linear(in_features=8, out_features=64, bias=False)
	(down_proj): Linear(in_features=64, out_features=8, bias=False)
	(act_fn): SiLU()
	)
	)
	(gate): MoEGate()
	(shared_experts): DeepseekV3MLP(
	(gate_proj): Linear(in_features=8, out_features=64, bias=False)
	(up_proj): Linear(in_features=8, out_features=64, bias=False)
	(down_proj): Linear(in_features=64, out_features=8, bias=False)
	(act_fn): SiLU()
	)
	)
	(input_layernorm): DeepseekV3RMSNorm()
	(post_attention_layernorm): DeepseekV3RMSNorm()
	)
	(2): DeepseekV3DecoderLayer(
	(self_attn): DeepseekV3Attention(
	(q_a_proj): Linear(in_features=8, out_features=32, bias=False)
	(q_a_layernorm): DeepseekV3RMSNorm()
	(q_b_proj): Linear(in_features=32, out_features=1024, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=8, out_features=576, bias=False)
	(kv_a_layernorm): DeepseekV3RMSNorm()
	(kv_b_proj): Linear(in_features=384, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=8, bias=False)
	(rotary_emb): DeepseekV3YarnRotaryEmbedding()
	)
	(mlp): DeepseekV3MoE(
	(experts): ModuleList(
	(0-31): 32 x DeepseekV3MLP(
	(gate_proj): Linear(in_features=8, out_features=64, bias=False)
	(up_proj): Linear(in_features=8, out_features=64, bias=False)
	(down_proj): Linear(in_features=64, out_features=8, bias=False)
	(act_fn): SiLU()
	)
	)
	(gate): MoEGate()
	(shared_experts): DeepseekV3MLP(
	(gate_proj): Linear(in_features=8, out_features=64, bias=False)
	(up_proj): Linear(in_features=8, out_features=64, bias=False)
	(down_proj): Linear(in_features=64, out_features=8, bias=False)
	(act_fn): SiLU()
	)
	)
	(input_layernorm): DeepseekV3RMSNorm()
	(post_attention_layernorm): DeepseekV3RMSNorm()
	(enorm): Glm4MoeRMSNorm((8,), eps=1e-06)
	(hnorm): Glm4MoeRMSNorm((8,), eps=1e-06)
	(eh_proj): Linear(in_features=16, out_features=8, bias=False)
	(shared_head): SharedHead(
	(norm): Glm4MoeRMSNorm((8,), eps=1e-06)
	(head): Linear(in_features=8, out_features=129280, bias=False)
	)
	(embed_tokens): Embedding(129280, 8)
	)
	)
	(norm): DeepseekV3RMSNorm()
	)
	(lm_head): Linear(in_features=8, out_features=129280, bias=False)
	)
	```