---
tags:
- int4
- vllm
- llm-compressor
- compressed-tensors
- conversational
license: other
license_name: "kimi-k3"
library_name: transformers
pipeline_tag: image-text-to-text
base_model: moonshotai/Kimi-K3
---

# Kimi-K3-INT4

## Model Overview

- **Model Architecture:** KimiK3ForConditionalGeneration
- **Input:** Text / Image
- **Output:** Text
- **Model Optimizations:**
  - **Weight quantization:** INT4
  - **Activation quantization:** None (original precision)
- **Release Date:** 2026-09-28
- **Version:** 1.0
- **Model Developers:** RedHatAI

This model is a quantized version of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3).

## Model Optimizations

This model was obtained by quantizing the weights of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) to signed INT4 with group size 128 while keeping activations in their original precision. The weights are stored in the packed `compressed-tensors` format for vLLM inference.

The quantized weight representation reduces the weight precision from 16 to 4 bits per parameter, reducing quantized weight storage and associated GPU memory requirements by approximately 75%.

The quantization was performed with [LLM Compressor](https://github.com/vllm-project/llm-compressor). The final checkpoint uses the `compressed-tensors` `pack-quantized` representation and is served by vLLM with its WNA16/MARLIN backend.

## Deployment

### Use with vLLM

Kimi K3 requires substantial hardware. The example below follows the upstream Kimi K3 serving guidance and the configuration used for validation of this checkpoint. Increase `--max-model-len` if the available memory and workload permit it.

```bash
vllm serve RedHatAI/Kimi-K3-INT4 \
  --tensor-parallel-size 8 \
  --enforce-eager \
  --trust-remote-code \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --enable-auto-tool-choice
```

The model always has reasoning enabled. For multi-turn conversations, preserve the complete assistant message, including `reasoning_content` and any tool calls, when passing the response back to the model.

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="RedHatAI/Kimi-K3-INT4",
    messages=[{"role": "user", "content": "Explain quantum mechanics clearly."}],
)

print(response.choices[0].message.content)
```

## Creation

The checkpoint was produced with layerwise compression and decompression. 

<details>

The core quantization configuration was:

```python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "lm_head",
        r"re:.*block_sparse_moe\.gate",
        r"re:.*vision_tower.*",
    ],
)

oneshot(
    model=model,
    tokenizer=processor.tokenizer,
    dataset="perfectblend",
    splits="train[:512]",
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
    trust_remote_code_model=True,
    pipeline="sequential",
    sequential_targets=["KimiDecoderLayer"],
    sequential_targets_per_subgraph=4,
    layerwise_decompression=True,
    layerwise_compression=True,
)
```

</details>

## Evaluation

This model was evaluated on  MATH-500 and GPQA Diamond using [lighteval](https://github.com/neuralmagic/lighteval), with the model served through vLLM's OpenAI-compatible API. The baseline was evaluated with the same protocol. These are preliminary smoke-test results, not full benchmark scores.

### Accuracy

<table>
  <thead>
    <tr>
      <th>Category</th>
      <th>Benchmark</th>
      <th>moonshotai/Kimi-K3</th>
      <th>RedHatAI/Kimi-K3-INT4</th>
      <th>Recovery</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td rowspan="2"><b>Reasoning</b></td>
      <td>MATH-500 (0-shot, pass@1</td>
      <td>90%</td>
      <td>87%</td>
      <td>97%</td>
    </tr>
    <tr>
      <td>GPQA Diamond (0-shot, pass@1)</td>
      <td>91%</td>
      <td>92%</td>
      <td>101%</td>
    </tr>
  </tbody>
</table>

## License

The model is provided under the [Kimi K3 license](https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE). Review the original model license and terms before use or redistribution.