Xiao-AMD commited on
Commit
3966522
·
verified ·
1 Parent(s): b3c64bd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +19 -10
README.md CHANGED
@@ -26,12 +26,22 @@ This model was built with the Kimi-K2.7-Code model by applying [AMD-Quark](https
26
 
27
  The model was quantized from [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) using [AMD-Quark](https://quark.docs.amd.com/latest/index.html). The MoE/Linear weights and activations are quantized to OCP MXFP4, while the attention projections use FP8 (E4M3). The vision tower and multimodal projector are kept at BF16.
28
 
29
- **Quantization scheme (from `config.json`):**
30
 
31
- - Weights: `fp4`, per-group (group size 32), `e8m0` scale, static
32
- - Activations: `fp4`, per-group (group size 32), `e8m0` scale, dynamic
33
- - Attention (`*self_attn*`): `fp8_e4m3`, per-channel, symmetric
34
- - Excludes: `re:.*mm_projector.*`, per-layer `mlp.gate`, vision encoder blocks
 
 
 
 
 
 
 
 
 
 
35
 
36
  # Deployment
37
  ### Use with vLLM
@@ -74,9 +84,9 @@ The model was evaluated on the GSM8K benchmark.
74
  <tr>
75
  <td>GSM8K (strict-match)
76
  </td>
77
- <td>0.9507
78
  </td>
79
- <td>0.9480
80
  </td>
81
  <td>99.7%
82
  </td>
@@ -84,9 +94,9 @@ The model was evaluated on the GSM8K benchmark.
84
  <tr>
85
  <td>GSM8K (flexible-extract)
86
  </td>
87
- <td>0.9515
88
  </td>
89
- <td>0.9477
90
  </td>
91
  <td>99.6%
92
  </td>
@@ -123,4 +133,3 @@ lm_eval --model local-completions \
123
 
124
  # License
125
  Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
126
-
 
26
 
27
  The model was quantized from [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) using [AMD-Quark](https://quark.docs.amd.com/latest/index.html). The MoE/Linear weights and activations are quantized to OCP MXFP4, while the attention projections use FP8 (E4M3). The vision tower and multimodal projector are kept at BF16.
28
 
29
+ **Quantization script:**
30
 
31
+ ```bash
32
+ cd Quark/examples/torch/language_modeling/llm_ptq/
33
+
34
+ python3 quantize_quark.py \
35
+ --model_dir moonshotai/Kimi-K2.7-Code \
36
+ --output_dir Kimi-K2.7-Code-MXFP4 \
37
+ --file2file_quantization \
38
+ --trust_remote_code \
39
+ --quant_scheme mxfp4 \
40
+ --layer_quant_scheme '*self_attn*' ptpc_fp8 \
41
+ --exclude_layers "*lm_head*" "*mlp.gate" "*mm_projector*" \
42
+ "*vision_tower*" "mtp.*" "*shared_expert_gate*" "*router*" \
43
+ --model_export hf_format
44
+ ```
45
 
46
  # Deployment
47
  ### Use with vLLM
 
84
  <tr>
85
  <td>GSM8K (strict-match)
86
  </td>
87
+ <td>95.07%
88
  </td>
89
+ <td>94.80%
90
  </td>
91
  <td>99.7%
92
  </td>
 
94
  <tr>
95
  <td>GSM8K (flexible-extract)
96
  </td>
97
+ <td>95.15%
98
  </td>
99
+ <td>94.77%
100
  </td>
101
  <td>99.6%
102
  </td>
 
133
 
134
  # License
135
  Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.