Instructions to use QuantTrio/Kimi-K3-Cubic-2.5Bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="QuantTrio/Kimi-K3-Cubic-2.5Bit", trust_remote_code=True)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("QuantTrio/Kimi-K3-Cubic-2.5Bit", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantTrio/Kimi-K3-Cubic-2.5Bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/QuantTrio/Kimi-K3-Cubic-2.5Bit

SGLang

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantTrio/Kimi-K3-Cubic-2.5Bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantTrio/Kimi-K3-Cubic-2.5Bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with Docker Model Runner:
```
docker model run hf.co/QuantTrio/Kimi-K3-Cubic-2.5Bit
```

Roadmap for DSpark support in the Cubic vLLM branch?

by bencswong2 - opened 1 day ago

Discussion

bencswong2

1 day ago

Thanks for the great work on the Cubic branch. I've been running some benchmarks with Kimi-K3-Cubic-2.5Bit and the results look very promising — both accuracy and throughput are quite good.

Is there a roadmap or estimated timeline for adding DSpark support to the Cubic vLLM branch? DSpark integration would be very useful for production deployment, and I'd be happy to help test early builds or provide feedback if needed.

Thanks!

bencswong2

1 day ago

•

edited 1 day ago

nerdctl run -d
--restart always
--name vllm-kimi-k3
--gpus '"device=0,1,2,3,4,5,6,7"'
--ipc=host
--ulimit memlock=-1
--ulimit stack=67108864
-v /bmcp_lvm_fs/cusa/models:/ssd1
-p 31001:31001
-e VLLM_CUBIC_DYNAMIC_A8=1
-e VLLM_USE_RUST_FRONTEND=1
-e VLLM_ENGINE_READY_TIMEOUT_S=1200
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
vllm/vllm-openai:v0.26.1-cubic-cu13-h20-pr51508
--model /ssd1/Kimi-K3-Cubic-2.5Bit
--quantization cubic
--kv-cache-dtype fp8
--served-model-name Kimi-K3
--max-num-seqs 10
--max-model-len auto
--max-num-batched-tokens 2048
--enable-prefix-caching
--host 0.0.0.0
--port 31001
--safetensors-load-strategy=prefetch
--attention-backend FLASHMLA
--moe-backend marlin
--enable-ep-weight-filter
--no-enable-flashinfer-autotune
--trust-remote-code
--tensor-parallel-size 8
--enable-expert-parallel
--tool-call-parser kimi_k3
--reasoning-parser kimi_k3
--enable-auto-tool-choice
--mm-encoder-tp-mode data
--gpu-memory-utilization 0.97
--disable-uvicorn-access-log
--no-async-scheduling
--disable-custom-all-reduce
--compilation-config '{"pass_config":{"fuse_allreduce_rms":false}}'
--speculative-config '{"model":"/ssd1/Kimi-K3-DSpark", "num_speculative_tokens":7, "method": "dspark", "draft_sample_method": "probabilistic", "rejection_sample_method": "block", "kv_cache_dtype": "bfloat16"}'

tclf90

QuantTrio org about 14 hours ago

Thanks for checking it out and testing!

Cubic is still in its very first iteration. I just finished putting together a short technical report (https://arxiv.org/html/2608.06763v1), and I’m now trying it on smaller models like Qwen3.5 and DeepSeek V4 Flash to make it more stable and general, with some throughput work still ongoing.

Once that’s done, I’ll come back to DSpark. I tried it briefly before and compatibility didn’t seem too difficult. Having a working DSpark setup to test against would be great. Thanks for the support!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment