Update README.md
Browse files
README.md
CHANGED
|
@@ -26,12 +26,22 @@ This model was built with the Kimi-K2.7-Code model by applying [AMD-Quark](https
|
|
| 26 |
|
| 27 |
The model was quantized from [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) using [AMD-Quark](https://quark.docs.amd.com/latest/index.html). The MoE/Linear weights and activations are quantized to OCP MXFP4, while the attention projections use FP8 (E4M3). The vision tower and multimodal projector are kept at BF16.
|
| 28 |
|
| 29 |
-
**Quantization
|
| 30 |
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
# Deployment
|
| 37 |
### Use with vLLM
|
|
@@ -74,9 +84,9 @@ The model was evaluated on the GSM8K benchmark.
|
|
| 74 |
<tr>
|
| 75 |
<td>GSM8K (strict-match)
|
| 76 |
</td>
|
| 77 |
-
<td>
|
| 78 |
</td>
|
| 79 |
-
<td>
|
| 80 |
</td>
|
| 81 |
<td>99.7%
|
| 82 |
</td>
|
|
@@ -84,9 +94,9 @@ The model was evaluated on the GSM8K benchmark.
|
|
| 84 |
<tr>
|
| 85 |
<td>GSM8K (flexible-extract)
|
| 86 |
</td>
|
| 87 |
-
<td>
|
| 88 |
</td>
|
| 89 |
-
<td>
|
| 90 |
</td>
|
| 91 |
<td>99.6%
|
| 92 |
</td>
|
|
@@ -123,4 +133,3 @@ lm_eval --model local-completions \
|
|
| 123 |
|
| 124 |
# License
|
| 125 |
Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
|
| 126 |
-
|
|
|
|
| 26 |
|
| 27 |
The model was quantized from [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) using [AMD-Quark](https://quark.docs.amd.com/latest/index.html). The MoE/Linear weights and activations are quantized to OCP MXFP4, while the attention projections use FP8 (E4M3). The vision tower and multimodal projector are kept at BF16.
|
| 28 |
|
| 29 |
+
**Quantization script:**
|
| 30 |
|
| 31 |
+
```bash
|
| 32 |
+
cd Quark/examples/torch/language_modeling/llm_ptq/
|
| 33 |
+
|
| 34 |
+
python3 quantize_quark.py \
|
| 35 |
+
--model_dir moonshotai/Kimi-K2.7-Code \
|
| 36 |
+
--output_dir Kimi-K2.7-Code-MXFP4 \
|
| 37 |
+
--file2file_quantization \
|
| 38 |
+
--trust_remote_code \
|
| 39 |
+
--quant_scheme mxfp4 \
|
| 40 |
+
--layer_quant_scheme '*self_attn*' ptpc_fp8 \
|
| 41 |
+
--exclude_layers "*lm_head*" "*mlp.gate" "*mm_projector*" \
|
| 42 |
+
"*vision_tower*" "mtp.*" "*shared_expert_gate*" "*router*" \
|
| 43 |
+
--model_export hf_format
|
| 44 |
+
```
|
| 45 |
|
| 46 |
# Deployment
|
| 47 |
### Use with vLLM
|
|
|
|
| 84 |
<tr>
|
| 85 |
<td>GSM8K (strict-match)
|
| 86 |
</td>
|
| 87 |
+
<td>95.07%
|
| 88 |
</td>
|
| 89 |
+
<td>94.80%
|
| 90 |
</td>
|
| 91 |
<td>99.7%
|
| 92 |
</td>
|
|
|
|
| 94 |
<tr>
|
| 95 |
<td>GSM8K (flexible-extract)
|
| 96 |
</td>
|
| 97 |
+
<td>95.15%
|
| 98 |
</td>
|
| 99 |
+
<td>94.77%
|
| 100 |
</td>
|
| 101 |
<td>99.6%
|
| 102 |
</td>
|
|
|
|
| 133 |
|
| 134 |
# License
|
| 135 |
Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
|
|
|