Instructions to use nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModel

processor = AutoProcessor.from_pretrained("nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4", trust_remote_code=True)
model = AutoModel.from_pretrained("nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4

SGLang

How to use nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4 with Docker Model Runner:
```
docker model run hf.co/nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4
```

Tiny random Kimi-K2.7-Code INT4

A tiny, randomly initialized Kimi-K2.7-Code checkpoint for testing checkpoint loading. It has two language layers and packed INT4 routed-expert weights. The dimensions are reduced, while the architecture, tensor naming, and packed-weight format follow the source model. Other weights are random BF16. This model is not useful for language generation or quality evaluation.

The packed expert weights can be decoded to BF16 during loading by QLab. This fixture was tested with Transformers 5.16.1 and trust_remote_code=True.

The configuration and Python model/tokenizer code are derived from moonshotai/Kimi-K2.7-Code. See LICENSE for the source license.

Downloads last month: 566

Safetensors

Model size

3.24M params

Tensor type

I32

BF16

Model tree for nikita-savelyev-cerebras/tiny-random-kimi-k2.7-code-int4

Base model

moonshotai/Kimi-K2.7-Code

Finetuned

(11)

this model