| --- |
| tags: |
| - int4 |
| - vllm |
| - llm-compressor |
| - compressed-tensors |
| - conversational |
| license: other |
| license_name: "kimi-k3" |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| base_model: moonshotai/Kimi-K3 |
| --- |
| |
| # Kimi-K3-INT4 |
|
|
| ## Model Overview |
|
|
| - **Model Architecture:** KimiK3ForConditionalGeneration |
| - **Input:** Text / Image |
| - **Output:** Text |
| - **Model Optimizations:** |
| - **Weight quantization:** INT4 |
| - **Activation quantization:** None (original precision) |
| - **Release Date:** 2026-09-28 |
| - **Version:** 1.0 |
| - **Model Developers:** RedHatAI |
|
|
| This model is a quantized version of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3). |
|
|
| ## Model Optimizations |
|
|
| This model was obtained by quantizing the weights of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) to signed INT4 with group size 128 while keeping activations in their original precision. The weights are stored in the packed `compressed-tensors` format for vLLM inference. |
|
|
| The quantized weight representation reduces the weight precision from 16 to 4 bits per parameter, reducing quantized weight storage and associated GPU memory requirements by approximately 75%. |
|
|
| The quantization was performed with [LLM Compressor](https://github.com/vllm-project/llm-compressor). The final checkpoint uses the `compressed-tensors` `pack-quantized` representation and is served by vLLM with its WNA16/MARLIN backend. |
|
|
| ## Deployment |
|
|
| ### Use with vLLM |
|
|
| Kimi K3 requires substantial hardware. The example below follows the upstream Kimi K3 serving guidance and the configuration used for validation of this checkpoint. Increase `--max-model-len` if the available memory and workload permit it. |
|
|
| ```bash |
| vllm serve RedHatAI/Kimi-K3-INT4 \ |
| --tensor-parallel-size 8 \ |
| --enforce-eager \ |
| --trust-remote-code \ |
| --reasoning-parser kimi_k3 \ |
| --tool-call-parser kimi_k3 \ |
| --enable-auto-tool-choice |
| ``` |
|
|
| The model always has reasoning enabled. For multi-turn conversations, preserve the complete assistant message, including `reasoning_content` and any tool calls, when passing the response back to the model. |
|
|
| ```python |
| from openai import OpenAI |
| |
| client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") |
| |
| response = client.chat.completions.create( |
| model="RedHatAI/Kimi-K3-INT4", |
| messages=[{"role": "user", "content": "Explain quantum mechanics clearly."}], |
| ) |
| |
| print(response.choices[0].message.content) |
| ``` |
|
|
| ## Creation |
|
|
| The checkpoint was produced with layerwise compression and decompression. |
|
|
| <details> |
|
|
| The core quantization configuration was: |
|
|
| ```python |
| from llmcompressor import oneshot |
| from llmcompressor.modifiers.quantization import QuantizationModifier |
| |
| recipe = QuantizationModifier( |
| targets="Linear", |
| scheme="W4A16", |
| ignore=[ |
| "lm_head", |
| r"re:.*block_sparse_moe\.gate", |
| r"re:.*vision_tower.*", |
| ], |
| ) |
| |
| oneshot( |
| model=model, |
| tokenizer=processor.tokenizer, |
| dataset="perfectblend", |
| splits="train[:512]", |
| recipe=recipe, |
| max_seq_length=2048, |
| num_calibration_samples=512, |
| trust_remote_code_model=True, |
| pipeline="sequential", |
| sequential_targets=["KimiDecoderLayer"], |
| sequential_targets_per_subgraph=4, |
| layerwise_decompression=True, |
| layerwise_compression=True, |
| ) |
| ``` |
|
|
| </details> |
|
|
| ## Evaluation |
|
|
| This model was evaluated on MATH-500 and GPQA Diamond using [lighteval](https://github.com/neuralmagic/lighteval), with the model served through vLLM's OpenAI-compatible API. The baseline was evaluated with the same protocol. These are preliminary smoke-test results, not full benchmark scores. |
|
|
| ### Accuracy |
|
|
| <table> |
| <thead> |
| <tr> |
| <th>Category</th> |
| <th>Benchmark</th> |
| <th>moonshotai/Kimi-K3</th> |
| <th>RedHatAI/Kimi-K3-INT4</th> |
| <th>Recovery</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td rowspan="2"><b>Reasoning</b></td> |
| <td>MATH-500 (0-shot, pass@1</td> |
| <td>90%</td> |
| <td>87%</td> |
| <td>97%</td> |
| </tr> |
| <tr> |
| <td>GPQA Diamond (0-shot, pass@1)</td> |
| <td>91%</td> |
| <td>92%</td> |
| <td>101%</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| ## License |
|
|
| The model is provided under the [Kimi K3 license](https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE). Review the original model license and terms before use or redistribution. |
|
|
