File size: 4,315 Bytes
e5a024d 6bea231 e5a024d d5048a3 e5a024d d5048a3 e5a024d d5048a3 e5a024d d5048a3 e5a024d 85284ef e5a024d d5048a3 e5a024d 2dba1e4 6c81b65 e5a024d 2dba1e4 6c81b65 e5a024d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | ---
tags:
- int4
- vllm
- llm-compressor
- compressed-tensors
- conversational
license: other
license_name: "kimi-k3"
library_name: transformers
pipeline_tag: image-text-to-text
base_model: moonshotai/Kimi-K3
---
# Kimi-K3-INT4
## Model Overview
- **Model Architecture:** KimiK3ForConditionalGeneration
- **Input:** Text / Image
- **Output:** Text
- **Model Optimizations:**
- **Weight quantization:** INT4
- **Activation quantization:** None (original precision)
- **Release Date:** 2026-09-28
- **Version:** 1.0
- **Model Developers:** RedHatAI
This model is a quantized version of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3).
## Model Optimizations
This model was obtained by quantizing the weights of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) to signed INT4 with group size 128 while keeping activations in their original precision. The weights are stored in the packed `compressed-tensors` format for vLLM inference.
The quantized weight representation reduces the weight precision from 16 to 4 bits per parameter, reducing quantized weight storage and associated GPU memory requirements by approximately 75%.
The quantization was performed with [LLM Compressor](https://github.com/vllm-project/llm-compressor). The final checkpoint uses the `compressed-tensors` `pack-quantized` representation and is served by vLLM with its WNA16/MARLIN backend.
## Deployment
### Use with vLLM
Kimi K3 requires substantial hardware. The example below follows the upstream Kimi K3 serving guidance and the configuration used for validation of this checkpoint. Increase `--max-model-len` if the available memory and workload permit it.
```bash
vllm serve RedHatAI/Kimi-K3-INT4 \
--tensor-parallel-size 8 \
--enforce-eager \
--trust-remote-code \
--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--enable-auto-tool-choice
```
The model always has reasoning enabled. For multi-turn conversations, preserve the complete assistant message, including `reasoning_content` and any tool calls, when passing the response back to the model.
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
model="RedHatAI/Kimi-K3-INT4",
messages=[{"role": "user", "content": "Explain quantum mechanics clearly."}],
)
print(response.choices[0].message.content)
```
## Creation
The checkpoint was produced with layerwise compression and decompression.
<details>
The core quantization configuration was:
```python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
recipe = QuantizationModifier(
targets="Linear",
scheme="W4A16",
ignore=[
"lm_head",
r"re:.*block_sparse_moe\.gate",
r"re:.*vision_tower.*",
],
)
oneshot(
model=model,
tokenizer=processor.tokenizer,
dataset="perfectblend",
splits="train[:512]",
recipe=recipe,
max_seq_length=2048,
num_calibration_samples=512,
trust_remote_code_model=True,
pipeline="sequential",
sequential_targets=["KimiDecoderLayer"],
sequential_targets_per_subgraph=4,
layerwise_decompression=True,
layerwise_compression=True,
)
```
</details>
## Evaluation
This model was evaluated on MATH-500 and GPQA Diamond using [lighteval](https://github.com/neuralmagic/lighteval), with the model served through vLLM's OpenAI-compatible API. The baseline was evaluated with the same protocol. These are preliminary smoke-test results, not full benchmark scores.
### Accuracy
<table>
<thead>
<tr>
<th>Category</th>
<th>Benchmark</th>
<th>moonshotai/Kimi-K3</th>
<th>RedHatAI/Kimi-K3-INT4</th>
<th>Recovery</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2"><b>Reasoning</b></td>
<td>MATH-500 (0-shot, pass@1</td>
<td>90%</td>
<td>87%</td>
<td>97%</td>
</tr>
<tr>
<td>GPQA Diamond (0-shot, pass@1)</td>
<td>91%</td>
<td>92%</td>
<td>101%</td>
</tr>
</tbody>
</table>
## License
The model is provided under the [Kimi K3 license](https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE). Review the original model license and terms before use or redistribution.
|