QuantTrio

/

Kimi-K3-Cubic-2.5Bit

like 3

Follow

QuantTrio 414

GB300 is probably not supported yet—there may still be a few CUDA kernel issues here and there.

Cubic Quant is very new. It’ll take some time to harden the kernels and gradually expand hardware support.

I’ll keep working through these compatibility issues. Thanks for testing it!

佈署成功了
我請kimi k2.7 幫忙debug的
我不是專業的 不知道它幹了什麼
但給你參考一下

     diff --git a/vllm/model_executor/layers/quantization/cubic_kernels.py b/vllm/model_executor/layers/quantization/cubic_kernels.py                                                                       
     index e9aa6e5c4..0988e4feb 100644                                                                                                                                                                      
     --- a/vllm/model_executor/layers/quantization/cubic_kernels.py                                                                                                                                         
     +++ b/vllm/model_executor/layers/quantization/cubic_kernels.py                                                                                                                                         
     @@ -7566,6 +7566,7 @@ def _launch_cubic_moe_situ_gemv_2bit(                                                                                                                                            
          *,                                                                                                                                                                                                
          logical_k: int,                                                                                                                                                                                   
          group_size: int,                                                                                                                                                                                  
     +    group_out: int = 1,                                                                                                                                                                               
          top_k: int,                                                                                                                                                                                       
          multiply_routed_weight: bool,                                                                                                                                                                     
          beta: float,                                                                                                                                                                                      
     @@ -7574,6 +7575,9 @@ def _launch_cubic_moe_situ_gemv_2bit(                                                                                                                                            
          route_ctas: int | None = None,                                                                                                                                                                    
          grouped_routes: int = 1,                                                                                                                                                                          
      ) -> None:                                                                                                                                                                                            
     +    assert group_out == 1, (                                                                                                                                                                          
     +        "Cubic SITU 2-bit MoE GEMV only supports group_out=1"                                                                                                                                         
     +    )                                                                                                                                                                                                 
          if dynamic_a8:                                                                                                                                                                                    
              kernel_inputs, input_scale = per_token_quant_int8(inputs.contiguous())                                                                                                                        
              ...                                                                                                                                                                                           
     @@ -9128,7 +9132,7 @@ def calibrate_cubic_moe_route_ctas(                                                                                                                                              
              )                                                                                                                                                                                             
                                                                                                                                                                                                            
              def launch(output=output, route_ctas=route_ctas) -> None:                                                                                                                                     
     -            if situ_beta is not None:                                                                                                                                                                 
     +            if situ_beta is not None and group_out == 1:                                                                                                                                              
                      _launch_cubic_moe_situ_gemv_2bit(                                                                                                                                                     
                          inputs,                                                                                                                                                                           
                          packed,                                                                                                                                                                           
     @@ -10554,6 +10558,7 @@ def cubic_fused_moe_dynamic_a8(                                                                                                                                                
              and num_bits == 2                                                                                                                                                                             
              and activation == MoEActivation.SITU                                                                                                                                                          
              and group_size in (128, 256, 512)                                                                                                                                                             
     +        and group_out == 1                                                                                                                                                                            
              and hidden_size % group_size == 0                                                                                                                                                             
          )                                                                                                                                                                                                 
          if fused_2bit_situ:                          

command:


vllm serve /data/k3_2_5 \
  --served-model-name Kimi-K3-2.5Bit \
  --trust-remote-code \
  --quantization cubic \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 4 \
  --disable-custom-all-reduce \
  --enable-expert-parallel \
  --mm-encoder-tp-mode data \
  --max-model-len auto \
  --max-num-seqs 5 \
  --max-num-batched-tokens 4096 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 8002 \
  --disable-uvicorn-access-log \
  --cpu-distributed-timeout-seconds 36000 \
  --attention-config '{"use_prefill_query_quantization": true, "mla_prefill_backend": "TRTLLM_RAGGED"}'

This is genuinely great stuff.
It’s likely the first quantization format that runs on vLLM.
Running Unsloth's GGUF on Llama is incredibly slow.
I wonder if "Cubic" quantization will eventually be merged into the main branch.
Also, have you run any benchmarks? I’m curious about the performance gap compared to the full-spec version.

Wow, increadible, I didn't expect it to be able to run on GB300 devices ATM.

Performance gap?

Accuracy wise, this is a parametric bit pack, so it's supposed to retain more accuracy than what Int and FP could, provided if the distribution is more monotonic-like (mostly hidden spaces that're learnt in deep learning models), technically speaking.
(This repo is quantized without data calibration. It could achieve better results when data from specified domains are provided for estimation.)

Though, I currently find it not working well with kv cache dtype set to cubic8 (maybe i had some bugs there), which needs more examinations later.
But anyway, having q (in qkv) unquantized is essential, so I temporarily made fp8_q16 dtype. This is a common practice, that one shouldn't quantize both query and key at the same time, or severe accuracy drop is to be expected. But then some people say KV shouldn't share the same quantization level, well then, it's also true, but ...

Some analytical details of Cubic Quant can be found here https://arxiv.org/abs/2608.06763

Model benchmarks, ... emm, I'm still trying to make these cuda kernels more robust, still working on cubic quant on deepseek v4 and qwen35.
But I'm running out of resources, so, Later I would say.

"I wonder if "Cubic" quantization will eventually be merged into the main branch." Later when things get ready.
But I heard them got lots of bot attacks lately, so temporarily, it is better that I would just fork one and build on it for now.

"GGUF" is good, once cubic quant is steady, we can try to make an alternative MoE kernel (yes to replace FusedMoE),
then we can just directly repack Unsloth's repo into cubic quant, and run with vLLM. But let's save it for later.