Instructions to use fxmeng/DSV2-Lite-IndexCache-Sparse-1250 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use fxmeng/DSV2-Lite-IndexCache-Sparse-1250 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="fxmeng/DSV2-Lite-IndexCache-Sparse-1250", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("fxmeng/DSV2-Lite-IndexCache-Sparse-1250", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use fxmeng/DSV2-Lite-IndexCache-Sparse-1250 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "fxmeng/DSV2-Lite-IndexCache-Sparse-1250"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fxmeng/DSV2-Lite-IndexCache-Sparse-1250",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/fxmeng/DSV2-Lite-IndexCache-Sparse-1250

SGLang

How to use fxmeng/DSV2-Lite-IndexCache-Sparse-1250 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "fxmeng/DSV2-Lite-IndexCache-Sparse-1250" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fxmeng/DSV2-Lite-IndexCache-Sparse-1250",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "fxmeng/DSV2-Lite-IndexCache-Sparse-1250" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fxmeng/DSV2-Lite-IndexCache-Sparse-1250",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use fxmeng/DSV2-Lite-IndexCache-Sparse-1250 with Docker Model Runner:
```
docker model run hf.co/fxmeng/DSV2-Lite-IndexCache-Sparse-1250
```

DSV2-Lite-IndexCache-Sparse-1250 / tokenization_deepseek_fast.py

Model export

Add validated sparse step 1250 model aligned to archived DSA recipe

1e3cf35 20 days ago

Raw History Blame Contribute Delete

1.37 kB

from typing import List, Optional, Union


from transformers.models.llama import LlamaTokenizerFast


class DeepseekTokenizerFast(LlamaTokenizerFast):

    def convert_ids_to_tokens(
        self, ids: Union[int, List[int]], skip_special_tokens: bool = False
    ) -> Union[str, List[str]]:
        """
        Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and
        added tokens.
        Args:
            ids (`int` or `List[int]`):
                The token id (or token ids) to convert to tokens.
            skip_special_tokens (`bool`, *optional*, defaults to `False`):
                Whether or not to remove special tokens in the decoding.
        Returns:
            `str` or `List[str]`: The decoded token(s).
        """
        if isinstance(ids, int):
            return self._convert_id_to_token(ids)
        tokens = []
        for index in ids:
            index = int(index)
            if skip_special_tokens and index in self.all_special_ids:
                continue
            token = self._tokenizer.id_to_token(index)
            tokens.append(token if token is not None else "")
        return tokens
    
    def _convert_id_to_token(self, index: int) -> Optional[str]:
        token = self._tokenizer.id_to_token(int(index))
        return token if token is not None else ""

	from typing import List, Optional, Union


	from transformers.models.llama import LlamaTokenizerFast


	class DeepseekTokenizerFast(LlamaTokenizerFast):

	def convert_ids_to_tokens(
	self, ids: Union[int, List[int]], skip_special_tokens: bool = False
	) -> Union[str, List[str]]:
	"""
	Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and
	added tokens.

	Args:
	ids (`int` or `List[int]`):
	The token id (or token ids) to convert to tokens.
	skip_special_tokens (`bool`, optional, defaults to `False`):
	Whether or not to remove special tokens in the decoding.

	Returns:
	`str` or `List[str]`: The decoded token(s).
	"""
	if isinstance(ids, int):
	return self._convert_id_to_token(ids)
	tokens = []
	for index in ids:
	index = int(index)
	if skip_special_tokens and index in self.all_special_ids:
	continue
	token = self._tokenizer.id_to_token(index)
	tokens.append(token if token is not None else "")
	return tokens

	def _convert_id_to_token(self, index: int) -> Optional[str]:
	token = self._tokenizer.id_to_token(int(index))
	return token if token is not None else ""