Hugging Face's logo

electroglyph
/
KnowLine-4B-Gen5.5-onnx

Text Classification
ONNX
onnxruntime
decision-model
qwen3.5
Model card Files
xet
 Community
KnowLine-4B-Gen5.5-onnx / export-cpu-int8.log
electroglyph's picture
electroglyph
Upload folder using huggingface_hub
a05d4d2 verified
Raw History Blame Contribute Delete
155 kB
Valid precision + execution provider combinations are: FP32 CPU, FP32 CUDA, FP16 CUDA, FP16 DML, FP16 TRT-RTX, BF16 CUDA, BF16 TRT-RTX, INT8 CPU, INT8 CUDA, INT8 WebGPU, INT4 CPU, INT4 CUDA, INT4 DML, INT4 WebGPU
Extra options: {'accuracy_level': '4', 'block_size': '32', 'algo_config': 'k_quant'}
GroupQueryAttention (GQA) is used in this model.
GroupQueryAttention (GQA) is used in this model.
Loading weights: 0%| | 0/426 [00:00<?, ?it/s] Loading weights: 100%|██████████| 426/426 [00:00<00:00, 21186.87it/s]
2026-11-06 12:49:28,775 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize model with k_quant algorithm...
2026-11-06 12:49:28,797 neural_compressor [WARNING] - Model size > 2GB. Please use model path instead of onnx model object to quantize
2026-11-06 12:49:28,871 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:49:33,569 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:49:36,040 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:49:38,569 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:49:54,793 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:23,110 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:26,833 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:28,683 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:30,821 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:39,318 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:43,363 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:47,076 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:48,862 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:51,040 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:50:58,897 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:02,904 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:07,259 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:09,364 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:17,095 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:21,157 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:24,896 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:26,705 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:28,865 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:36,600 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:40,593 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:44,257 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:46,031 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:48,064 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:51:56,456 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:00,502 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:04,216 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:06,102 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:08,247 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:16,054 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:20,093 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:24,200 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:26,401 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:36,387 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:40,407 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:44,088 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:45,866 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:47,771 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:55,346 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:52:59,257 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:02,717 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:04,581 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:06,793 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:20,257 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:24,428 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:28,181 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:30,065 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:32,042 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:39,820 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:43,699 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:48,631 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:53:50,843 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:06,769 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:10,874 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:14,620 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:16,502 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:18,152 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:25,839 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:29,608 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:34,406 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:36,206 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:38,455 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:55,621 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:54:59,603 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:03,326 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:05,043 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:07,256 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:14,819 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:20,520 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:26,685 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:28,839 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:41,594 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:45,629 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:49,267 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:51,063 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:55:53,179 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:01,328 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:06,683 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:10,648 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:12,631 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:14,672 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:24,370 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:28,238 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:31,964 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:33,722 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:35,920 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:50,520 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:54,556 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:56:58,954 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:01,009 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:08,971 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:12,867 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:16,630 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:18,450 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:20,465 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:30,868 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:34,869 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:38,586 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:40,347 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:42,336 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:50,053 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:53,989 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:57,660 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:57:59,320 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:01,204 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:09,518 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:13,523 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:17,829 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:20,001 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:27,751 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:31,699 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:35,317 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:36,850 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:38,743 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:47,057 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:51,031 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:54,638 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:56,508 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:58:58,494 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:06,196 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:10,127 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:13,722 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:15,402 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:17,704 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:28,176 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:32,275 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:36,526 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:38,667 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:46,319 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:50,279 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:53,868 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:55,568 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 12:59:57,690 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:07,764 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:11,809 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:15,420 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:17,280 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:19,349 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
Reading embedding layer
Reading layer 0
Reading layer 1
Reading layer 2
Reading layer 3
Reading layer 4
Reading layer 5
Reading layer 6
Reading layer 7
Reading layer 8
Reading layer 9
Reading layer 10
Reading layer 11
Reading layer 12
Reading layer 13
Reading layer 14
Reading layer 15
Reading layer 16
Reading layer 17
Reading layer 18
Reading layer 19
Reading layer 20
Reading layer 21
Reading layer 22
Reading layer 23
Reading layer 24
Reading layer 25
Reading layer 26
Reading layer 27
Reading layer 28
Reading layer 29
Reading layer 30
Reading layer 31
Reading final norm
Reading LM head
Building MTP (multi-token prediction) head -> mtp.onnx
Saving ONNX model in model-cpu-int8
Progress: [ ] 0.50% Progress: [ ] 1.00% Progress: [ ] 1.49% Progress: [ ] 1.99% Progress: [ ] 2.49% Progress: [ ] 2.99% Progress: [ ] 3.48% Progress: [ ] 3.98% Progress: [ ] 4.48% Progress: [ ] 4.98% Progress: [# ] 5.47% Progress: [# ] 5.97% Progress: [# ] 6.47% Progress: [# ] 6.97% Progress: [# ] 7.46% Progress: [# ] 7.96% Progress: [# ] 8.46% Progress: [# ] 8.96% Progress: [# ] 9.45% Progress: [# ] 9.95% Progress: [## ] 10.45% Progress: [## ] 10.95% Progress: [## ] 11.44% Progress: [## ] 11.94% Progress: [## ] 12.44% Progress: [## ] 12.94% Progress: [## ] 13.43% Progress: [## ] 13.93% Progress: [## ] 14.43% Progress: [## ] 14.93% Progress: [### ] 15.42% Progress: [### ] 15.92% Progress: [### ] 16.42% Progress: [### ] 16.92% Progress: [### ] 17.41% Progress: [### ] 17.91% Progress: [### ] 18.41% Progress: [### ] 18.91% Progress: [### ] 19.40% Progress: [### ] 19.90% Progress: [#### ] 20.40% Progress: [#### ] 20.90% Progress: [#### ] 21.39% Progress: [#### ] 21.89% Progress: [#### ] 22.39% Progress: [#### ] 22.89% Progress: [#### ] 23.38% Progress: [#### ] 23.88% Progress: [#### ] 24.38% Progress: [#### ] 24.88% Progress: [##### ] 25.37% Progress: [##### ] 25.87% Progress: [##### ] 26.37% Progress: [##### ] 26.87% Progress: [##### ] 27.36% Progress: [##### ] 27.86% Progress: [##### ] 28.36% Progress: [##### ] 28.86% Progress: [##### ] 29.35% Progress: [##### ] 29.85% Progress: [###### ] 30.35% Progress: [###### ] 30.85% Progress: [###### ] 31.34% Progress: [###### ] 31.84% Progress: [###### ] 32.34% Progress: [###### ] 32.84% Progress: [###### ] 33.33% Progress: [###### ] 33.83% Progress: [###### ] 34.33% Progress: [###### ] 34.83% Progress: [####### ] 35.32% Progress: [####### ] 35.82% Progress: [####### ] 36.32% Progress: [####### ] 36.82% Progress: [####### ] 37.31% Progress: [####### ] 37.81% Progress: [####### ] 38.31% Progress: [####### ] 38.81% Progress: [####### ] 39.30% Progress: [####### ] 39.80% Progress: [######## ] 40.30% Progress: [######## ] 40.80% Progress: [######## ] 41.29% Progress: [######## ] 41.79% Progress: [######## ] 42.29% Progress: [######## ] 42.79% Progress: [######## ] 43.28% Progress: [######## ] 43.78% Progress: [######## ] 44.28% Progress: [######## ] 44.78% Progress: [######### ] 45.27% Progress: [######### ] 45.77% Progress: [######### ] 46.27% Progress: [######### ] 46.77% Progress: [######### ] 47.26% Progress: [######### ] 47.76% Progress: [######### ] 48.26% Progress: [######### ] 48.76% Progress: [######### ] 49.25% Progress: [######### ] 49.75% Progress: [########## ] 50.25% Progress: [########## ] 50.75% Progress: [########## ] 51.24% Progress: [########## ] 51.74% Progress: [########## ] 52.24% Progress: [########## ] 52.74% Progress: [########## ] 53.23% Progress: [########## ] 53.73% Progress: [########## ] 54.23% Progress: [########## ] 54.73% Progress: [########### ] 55.22% Progress: [########### ] 55.72% Progress: [########### ] 56.22% Progress: [########### ] 56.72% Progress: [########### ] 57.21% Progress: [########### ] 57.71% Progress: [########### ] 58.21% Progress: [########### ] 58.71% Progress: [########### ] 59.20% Progress: [########### ] 59.70% Progress: [############ ] 60.20% Progress: [############ ] 60.70% Progress: [############ ] 61.19% Progress: [############ ] 61.69% Progress: [############ ] 62.19% Progress: [############ ] 62.69% Progress: [############ ] 63.18% Progress: [############ ] 63.68% Progress: [############ ] 64.18% Progress: [############ ] 64.68% Progress: [############# ] 65.17% Progress: [############# ] 65.67% Progress: [############# ] 66.17% Progress: [############# ] 66.67% Progress: [############# ] 67.16% Progress: [############# ] 67.66% Progress: [############# ] 68.16% Progress: [############# ] 68.66% Progress: [############# ] 69.15% Progress: [############# ] 69.65% Progress: [############## ] 70.15% Progress: [############## ] 70.65% Progress: [############## ] 71.14% Progress: [############## ] 71.64% Progress: [############## ] 72.14% Progress: [############## ] 72.64% Progress: [############## ] 73.13% Progress: [############## ] 73.63% Progress: [############## ] 74.13% Progress: [############## ] 74.63% Progress: [############### ] 75.12% Progress: [############### ] 75.62% Progress: [############### ] 76.12% Progress: [############### ] 76.62% Progress: [############### ] 77.11% Progress: [############### ] 77.61% Progress: [############### ] 78.11% Progress: [############### ] 78.61% Progress: [############### ] 79.10% Progress: [############### ] 79.60% Progress: [################ ] 80.10% Progress: [################ ] 80.60% Progress: [################ ] 81.09% Progress: [################ ] 81.59% Progress: [################ ] 82.09% Progress: [################ ] 82.59% Progress: [################ ] 83.08% Progress: [################ ] 83.58% Progress: [################ ] 84.08% Progress: [################ ] 84.58% Progress: [################# ] 85.07% Progress: [################# ] 85.57% Progress: [################# ] 86.07% Progress: [################# ] 86.57% Progress: [################# ] 87.06% Progress: [################# ] 87.56% Progress: [################# ] 88.06% Progress: [################# ] 88.56% Progress: [################# ] 89.05% Progress: [################# ] 89.55% Progress: [################## ] 90.05% Progress: [################## ] 90.55% Progress: [################## ] 91.04% Progress: [################## ] 91.54% Progress: [################## ] 92.04% Progress: [################## ] 92.54% Progress: [################## ] 93.03% Progress: [################## ] 93.53% Progress: [################## ] 94.03%2026-11-06 13:00:27,016 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:30,805 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:34,230 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:36,079 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:37,964 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:47,954 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:51,824 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:56,150 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:00:58,172 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:01:05,827 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:01:13,311 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:06:13,971 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of model with k_quant algorithm.
Progress: [################## ] 94.53% Progress: [################### ] 95.02% Progress: [################### ] 95.52% Progress: [################### ] 96.02% Progress: [################### ] 96.52% Progress: [################### ] 97.01% Progress: [################### ] 97.51% Progress: [################### ] 98.01% Progress: [################### ] 98.51% Progress: [################### ] 99.00% Progress: [################### ] 99.50% Progress: [####################] 100.00% 0it [00:00, ?it/s] Saving model.embed_tokens.weight_Q8 (u8, [248320,2560]): 0%| | 1/712 [00:00<00:10, 65.09it/s] Saving model.embed_tokens.weight_Q8 (u8, [248320,2560]): 0%| | 2/712 [00:00<04:25, 2.68it/s] Saving model.embed_tokens.weight_scales (f32, [248320,80]): 0%| | 2/712 [00:00<04:25, 2.68it/s] Saving model.embed_tokens.weight_scales (f32, [248320,80]): 0%| | 3/712 [00:00<03:36, 3.27it/s] Saving model.layers.0.input_layernorm.weight (f32, [2560]): 0%| | 3/712 [00:00<03:36, 3.27it/s] Saving model.layers.0.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 1%| | 4/712 [00:00<03:36, 3.27it/s] Saving model.layers.0.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 1%| | 5/712 [00:00<03:36, 3.27it/s] Saving model.layers.0.linear_attn.conv1d.weight (f32, [8192,1,4]): 1%| | 6/712 [00:00<03:35, 3.27it/s] Saving model.layers.0.linear_attn.conv1d.bias (f32, [8192]): 1%| | 7/712 [00:00<03:35, 3.27it/s] Saving model.layers.0.linear_attn.dt_bias (f32, [32]): 1%| | 8/712 [00:00<03:35, 3.27it/s] Saving model.layers.0.linear_attn.neg_exp_A (f32, [32]): 1%|▏ | 9/712 [00:00<03:35, 3.27it/s] Saving model.layers.0.linear_attn.norm.weight (f32, [128]): 1%|▏ | 10/712 [00:00<03:34, 3.27it/s] Saving model.layers.0.post_attention_layernorm.weight (f32, [2560]): 2%|▏ | 11/712 [00:00<03:34, 3.27it/s] Saving model.layers.1.input_layernorm.weight (f32, [2560]): 2%|▏ | 12/712 [00:00<03:34, 3.27it/s] Saving model.layers.1.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 2%|▏ | 13/712 [00:00<03:33, 3.27it/s] Saving model.layers.1.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 2%|▏ | 14/712 [00:00<03:33, 3.27it/s] Saving model.layers.1.linear_attn.conv1d.weight (f32, [8192,1,4]): 2%|▏ | 15/712 [00:00<03:33, 3.27it/s] Saving model.layers.1.linear_attn.conv1d.bias (f32, [8192]): 2%|▏ | 16/712 [00:00<03:32, 3.27it/s] Saving model.layers.1.linear_attn.dt_bias (f32, [32]): 2%|▏ | 17/712 [00:00<03:32, 3.27it/s] Saving model.layers.1.linear_attn.neg_exp_A (f32, [32]): 3%|▎ | 18/712 [00:00<03:32, 3.27it/s] Saving model.layers.1.linear_attn.norm.weight (f32, [128]): 3%|▎ | 19/712 [00:00<03:31, 3.27it/s] Saving model.layers.1.post_attention_layernorm.weight (f32, [2560]): 3%|▎ | 20/712 [00:00<03:31, 3.27it/s] Saving model.layers.2.input_layernorm.weight (f32, [2560]): 3%|▎ | 21/712 [00:00<03:31, 3.27it/s] Saving model.layers.2.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 3%|▎ | 22/712 [00:00<03:31, 3.27it/s] Saving model.layers.2.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 3%|▎ | 23/712 [00:00<03:30, 3.27it/s] Saving model.layers.2.linear_attn.conv1d.weight (f32, [8192,1,4]): 3%|▎ | 24/712 [00:00<03:30, 3.27it/s] Saving model.layers.2.linear_attn.conv1d.bias (f32, [8192]): 4%|▎ | 25/712 [00:00<03:30, 3.27it/s] Saving model.layers.2.linear_attn.dt_bias (f32, [32]): 4%|▎ | 26/712 [00:00<03:29, 3.27it/s] Saving model.layers.2.linear_attn.neg_exp_A (f32, [32]): 4%|▍ | 27/712 [00:00<03:29, 3.27it/s] Saving model.layers.2.linear_attn.norm.weight (f32, [128]): 4%|▍ | 28/712 [00:00<03:29, 3.27it/s] Saving model.layers.2.post_attention_layernorm.weight (f32, [2560]): 4%|▍ | 29/712 [00:00<03:28, 3.27it/s] Saving model.layers.3.input_layernorm.weight (f32, [2560]): 4%|▍ | 30/712 [00:00<03:28, 3.27it/s] Saving model.layers.3.attn.q_norm.layernorm.weight (f32, [256]): 4%|▍ | 31/712 [00:00<03:28, 3.27it/s] Saving model.layers.3.attn.k_norm.layernorm.weight (f32, [256]): 4%|▍ | 32/712 [00:00<03:27, 3.27it/s] Saving cos_cache (f32, [262144,32]): 5%|▍ | 33/712 [00:00<03:27, 3.27it/s] Saving sin_cache (f32, [262144,32]): 5%|▍ | 34/712 [00:01<03:27, 3.27it/s] Saving model.layers.3.post_attention_layernorm.weight (f32, [2560]): 5%|▍ | 35/712 [00:01<03:27, 3.27it/s] Saving model.layers.4.input_layernorm.weight (f32, [2560]): 5%|▌ | 36/712 [00:01<03:26, 3.27it/s] Saving model.layers.4.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 5%|▌ | 37/712 [00:01<03:26, 3.27it/s] Saving model.layers.4.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 5%|▌ | 38/712 [00:01<03:26, 3.27it/s] Saving model.layers.4.linear_attn.conv1d.weight (f32, [8192,1,4]): 5%|▌ | 39/712 [00:01<03:25, 3.27it/s] Saving model.layers.4.linear_attn.conv1d.weight (f32, [8192,1,4]): 6%|▌ | 40/712 [00:01<00:10, 62.98it/s] Saving model.layers.4.linear_attn.conv1d.bias (f32, [8192]): 6%|▌ | 40/712 [00:01<00:10, 62.98it/s] Saving model.layers.4.linear_attn.dt_bias (f32, [32]): 6%|▌ | 41/712 [00:01<00:10, 62.98it/s] Saving model.layers.4.linear_attn.neg_exp_A (f32, [32]): 6%|▌ | 42/712 [00:01<00:10, 62.98it/s] Saving model.layers.4.linear_attn.norm.weight (f32, [128]): 6%|▌ | 43/712 [00:01<00:10, 62.98it/s] Saving model.layers.4.post_attention_layernorm.weight (f32, [2560]): 6%|▌ | 44/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.input_layernorm.weight (f32, [2560]): 6%|▋ | 45/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 6%|▋ | 46/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 7%|▋ | 47/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.linear_attn.conv1d.weight (f32, [8192,1,4]): 7%|▋ | 48/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.linear_attn.conv1d.bias (f32, [8192]): 7%|▋ | 49/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.linear_attn.dt_bias (f32, [32]): 7%|▋ | 50/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.linear_attn.neg_exp_A (f32, [32]): 7%|▋ | 51/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.linear_attn.norm.weight (f32, [128]): 7%|▋ | 52/712 [00:01<00:10, 62.98it/s] Saving model.layers.5.post_attention_layernorm.weight (f32, [2560]): 7%|▋ | 53/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.input_layernorm.weight (f32, [2560]): 8%|▊ | 54/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 8%|▊ | 55/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 8%|▊ | 56/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.linear_attn.conv1d.weight (f32, [8192,1,4]): 8%|▊ | 57/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.linear_attn.conv1d.bias (f32, [8192]): 8%|▊ | 58/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.linear_attn.dt_bias (f32, [32]): 8%|▊ | 59/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.linear_attn.neg_exp_A (f32, [32]): 8%|▊ | 60/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.linear_attn.norm.weight (f32, [128]): 9%|▊ | 61/712 [00:01<00:10, 62.98it/s] Saving model.layers.6.post_attention_layernorm.weight (f32, [2560]): 9%|▊ | 62/712 [00:01<00:10, 62.98it/s] Saving model.layers.7.input_layernorm.weight (f32, [2560]): 9%|▉ | 63/712 [00:01<00:10, 62.98it/s] Saving model.layers.7.attn.q_norm.layernorm.weight (f32, [256]): 9%|▉ | 64/712 [00:01<00:10, 62.98it/s] Saving model.layers.7.attn.k_norm.layernorm.weight (f32, [256]): 9%|▉ | 65/712 [00:01<00:10, 62.98it/s] Saving model.layers.7.post_attention_layernorm.weight (f32, [2560]): 9%|▉ | 66/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.input_layernorm.weight (f32, [2560]): 9%|▉ | 67/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 10%|▉ | 68/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 10%|▉ | 69/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.linear_attn.conv1d.weight (f32, [8192,1,4]): 10%|▉ | 70/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.linear_attn.conv1d.bias (f32, [8192]): 10%|▉ | 71/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.linear_attn.dt_bias (f32, [32]): 10%|█ | 72/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.linear_attn.neg_exp_A (f32, [32]): 10%|█ | 73/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.linear_attn.norm.weight (f32, [128]): 10%|█ | 74/712 [00:01<00:10, 62.98it/s] Saving model.layers.8.post_attention_layernorm.weight (f32, [2560]): 11%|█ | 75/712 [00:01<00:10, 62.98it/s] Saving model.layers.9.input_layernorm.weight (f32, [2560]): 11%|█ | 76/712 [00:01<00:10, 62.98it/s] Saving model.layers.9.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 11%|█ | 77/712 [00:01<00:10, 62.98it/s] Saving model.layers.9.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 11%|█ | 78/712 [00:01<00:10, 62.98it/s] Saving model.layers.9.linear_attn.conv1d.weight (f32, [8192,1,4]): 11%|█ | 79/712 [00:01<00:10, 62.98it/s] Saving model.layers.9.linear_attn.conv1d.bias (f32, [8192]): 11%|█ | 80/712 [00:01<00:10, 62.98it/s] Saving model.layers.9.linear_attn.dt_bias (f32, [32]): 11%|█▏ | 81/712 [00:01<00:10, 62.98it/s] Saving model.layers.9.linear_attn.neg_exp_A (f32, [32]): 12%|█▏ | 82/712 [00:01<00:10, 62.98it/s] Saving model.layers.9.linear_attn.norm.weight (f32, [128]): 12%|█▏ | 83/712 [00:01<00:09, 62.98it/s] Saving model.layers.9.post_attention_layernorm.weight (f32, [2560]): 12%|█▏ | 84/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.input_layernorm.weight (f32, [2560]): 12%|█▏ | 85/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 12%|█▏ | 86/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 12%|█▏ | 87/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.linear_attn.conv1d.weight (f32, [8192,1,4]): 12%|█▏ | 88/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.linear_attn.conv1d.bias (f32, [8192]): 12%|█▎ | 89/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.linear_attn.dt_bias (f32, [32]): 13%|█▎ | 90/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.linear_attn.neg_exp_A (f32, [32]): 13%|█▎ | 91/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.linear_attn.norm.weight (f32, [128]): 13%|█▎ | 92/712 [00:01<00:09, 62.98it/s] Saving model.layers.10.post_attention_layernorm.weight (f32, [2560]): 13%|█▎ | 93/712 [00:01<00:09, 62.98it/s] Saving model.layers.11.input_layernorm.weight (f32, [2560]): 13%|█▎ | 94/712 [00:01<00:09, 62.98it/s] Saving model.layers.11.attn.q_norm.layernorm.weight (f32, [256]): 13%|█▎ | 95/712 [00:01<00:09, 62.98it/s] Saving model.layers.11.attn.k_norm.layernorm.weight (f32, [256]): 13%|█▎ | 96/712 [00:01<00:09, 62.98it/s] Saving model.layers.11.post_attention_layernorm.weight (f32, [2560]): 14%|█▎ | 97/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.input_layernorm.weight (f32, [2560]): 14%|█▍ | 98/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 14%|█▍ | 99/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 14%|█▍ | 100/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.linear_attn.conv1d.weight (f32, [8192,1,4]): 14%|█▍ | 101/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.linear_attn.conv1d.bias (f32, [8192]): 14%|█▍ | 102/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.linear_attn.dt_bias (f32, [32]): 14%|█▍ | 103/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.linear_attn.neg_exp_A (f32, [32]): 15%|█▍ | 104/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.linear_attn.norm.weight (f32, [128]): 15%|█▍ | 105/712 [00:01<00:09, 62.98it/s] Saving model.layers.12.post_attention_layernorm.weight (f32, [2560]): 15%|█▍ | 106/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.input_layernorm.weight (f32, [2560]): 15%|█▌ | 107/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 15%|█▌ | 108/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 15%|█▌ | 109/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.linear_attn.conv1d.weight (f32, [8192,1,4]): 15%|█▌ | 110/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.linear_attn.conv1d.bias (f32, [8192]): 16%|█▌ | 111/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.linear_attn.dt_bias (f32, [32]): 16%|█▌ | 112/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.linear_attn.neg_exp_A (f32, [32]): 16%|█▌ | 113/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.linear_attn.norm.weight (f32, [128]): 16%|█▌ | 114/712 [00:01<00:09, 62.98it/s] Saving model.layers.13.post_attention_layernorm.weight (f32, [2560]): 16%|█▌ | 115/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.input_layernorm.weight (f32, [2560]): 16%|█▋ | 116/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 16%|█▋ | 117/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 17%|█▋ | 118/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.linear_attn.conv1d.weight (f32, [8192,1,4]): 17%|█▋ | 119/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.linear_attn.conv1d.bias (f32, [8192]): 17%|█▋ | 120/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.linear_attn.dt_bias (f32, [32]): 17%|█▋ | 121/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.linear_attn.neg_exp_A (f32, [32]): 17%|█▋ | 122/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.linear_attn.norm.weight (f32, [128]): 17%|█▋ | 123/712 [00:01<00:09, 62.98it/s] Saving model.layers.14.post_attention_layernorm.weight (f32, [2560]): 17%|█▋ | 124/712 [00:01<00:09, 62.98it/s] Saving model.layers.15.input_layernorm.weight (f32, [2560]): 18%|█▊ | 125/712 [00:01<00:09, 62.98it/s] Saving model.layers.15.attn.q_norm.layernorm.weight (f32, [256]): 18%|█▊ | 126/712 [00:01<00:09, 62.98it/s] Saving model.layers.15.attn.k_norm.layernorm.weight (f32, [256]): 18%|█▊ | 127/712 [00:01<00:09, 62.98it/s] Saving model.layers.15.post_attention_layernorm.weight (f32, [2560]): 18%|█▊ | 128/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.input_layernorm.weight (f32, [2560]): 18%|█▊ | 129/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 18%|█▊ | 130/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 18%|█▊ | 131/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.linear_attn.conv1d.weight (f32, [8192,1,4]): 19%|█▊ | 132/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.linear_attn.conv1d.bias (f32, [8192]): 19%|█▊ | 133/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.linear_attn.dt_bias (f32, [32]): 19%|█▉ | 134/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.linear_attn.neg_exp_A (f32, [32]): 19%|█▉ | 135/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.linear_attn.norm.weight (f32, [128]): 19%|█▉ | 136/712 [00:01<00:09, 62.98it/s] Saving model.layers.16.post_attention_layernorm.weight (f32, [2560]): 19%|█▉ | 137/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.input_layernorm.weight (f32, [2560]): 19%|█▉ | 138/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 20%|█▉ | 139/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 20%|█▉ | 140/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.linear_attn.conv1d.weight (f32, [8192,1,4]): 20%|█▉ | 141/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.linear_attn.conv1d.bias (f32, [8192]): 20%|█▉ | 142/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.linear_attn.dt_bias (f32, [32]): 20%|██ | 143/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.linear_attn.neg_exp_A (f32, [32]): 20%|██ | 144/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.linear_attn.norm.weight (f32, [128]): 20%|██ | 145/712 [00:01<00:09, 62.98it/s] Saving model.layers.17.post_attention_layernorm.weight (f32, [2560]): 21%|██ | 146/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.input_layernorm.weight (f32, [2560]): 21%|██ | 147/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 21%|██ | 148/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 21%|██ | 149/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.linear_attn.conv1d.weight (f32, [8192,1,4]): 21%|██ | 150/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.linear_attn.conv1d.bias (f32, [8192]): 21%|██ | 151/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.linear_attn.dt_bias (f32, [32]): 21%|██▏ | 152/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.linear_attn.neg_exp_A (f32, [32]): 21%|██▏ | 153/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.linear_attn.norm.weight (f32, [128]): 22%|██▏ | 154/712 [00:01<00:08, 62.98it/s] Saving model.layers.18.post_attention_layernorm.weight (f32, [2560]): 22%|██▏ | 155/712 [00:01<00:08, 62.98it/s] Saving model.layers.19.input_layernorm.weight (f32, [2560]): 22%|██▏ | 156/712 [00:01<00:08, 62.98it/s] Saving model.layers.19.attn.q_norm.layernorm.weight (f32, [256]): 22%|██▏ | 157/712 [00:01<00:08, 62.98it/s] Saving model.layers.19.attn.k_norm.layernorm.weight (f32, [256]): 22%|██▏ | 158/712 [00:01<00:08, 62.98it/s] Saving model.layers.19.post_attention_layernorm.weight (f32, [2560]): 22%|██▏ | 159/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.input_layernorm.weight (f32, [2560]): 22%|██▏ | 160/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 23%|██▎ | 161/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 23%|██▎ | 162/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.linear_attn.conv1d.weight (f32, [8192,1,4]): 23%|██▎ | 163/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.linear_attn.conv1d.bias (f32, [8192]): 23%|██▎ | 164/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.linear_attn.dt_bias (f32, [32]): 23%|██▎ | 165/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.linear_attn.neg_exp_A (f32, [32]): 23%|██▎ | 166/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.linear_attn.norm.weight (f32, [128]): 23%|██▎ | 167/712 [00:01<00:08, 62.98it/s] Saving model.layers.20.post_attention_layernorm.weight (f32, [2560]): 24%|██▎ | 168/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.input_layernorm.weight (f32, [2560]): 24%|██▎ | 169/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 24%|██▍ | 170/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 24%|██▍ | 171/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.linear_attn.conv1d.weight (f32, [8192,1,4]): 24%|██▍ | 172/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.linear_attn.conv1d.bias (f32, [8192]): 24%|██▍ | 173/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.linear_attn.dt_bias (f32, [32]): 24%|██▍ | 174/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.linear_attn.neg_exp_A (f32, [32]): 25%|██▍ | 175/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.linear_attn.norm.weight (f32, [128]): 25%|██▍ | 176/712 [00:01<00:08, 62.98it/s] Saving model.layers.21.post_attention_layernorm.weight (f32, [2560]): 25%|██▍ | 177/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.input_layernorm.weight (f32, [2560]): 25%|██▌ | 178/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 25%|██▌ | 179/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 25%|██▌ | 180/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.linear_attn.conv1d.weight (f32, [8192,1,4]): 25%|██▌ | 181/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.linear_attn.conv1d.bias (f32, [8192]): 26%|██▌ | 182/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.linear_attn.dt_bias (f32, [32]): 26%|██▌ | 183/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.linear_attn.neg_exp_A (f32, [32]): 26%|██▌ | 184/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.linear_attn.norm.weight (f32, [128]): 26%|██▌ | 185/712 [00:01<00:08, 62.98it/s] Saving model.layers.22.post_attention_layernorm.weight (f32, [2560]): 26%|██▌ | 186/712 [00:01<00:08, 62.98it/s] Saving model.layers.23.input_layernorm.weight (f32, [2560]): 26%|██▋ | 187/712 [00:01<00:08, 62.98it/s] Saving model.layers.23.attn.q_norm.layernorm.weight (f32, [256]): 26%|██▋ | 188/712 [00:01<00:08, 62.98it/s] Saving model.layers.23.attn.k_norm.layernorm.weight (f32, [256]): 27%|██▋ | 189/712 [00:01<00:08, 62.98it/s] Saving model.layers.23.post_attention_layernorm.weight (f32, [2560]): 27%|██▋ | 190/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.input_layernorm.weight (f32, [2560]): 27%|██▋ | 191/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 27%|██▋ | 192/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 27%|██▋ | 193/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.linear_attn.conv1d.weight (f32, [8192,1,4]): 27%|██▋ | 194/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.linear_attn.conv1d.bias (f32, [8192]): 27%|██▋ | 195/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.linear_attn.dt_bias (f32, [32]): 28%|██▊ | 196/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.linear_attn.neg_exp_A (f32, [32]): 28%|██▊ | 197/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.linear_attn.norm.weight (f32, [128]): 28%|██▊ | 198/712 [00:01<00:08, 62.98it/s] Saving model.layers.24.post_attention_layernorm.weight (f32, [2560]): 28%|██▊ | 199/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.input_layernorm.weight (f32, [2560]): 28%|██▊ | 200/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 28%|██▊ | 201/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 28%|██▊ | 202/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.linear_attn.conv1d.weight (f32, [8192,1,4]): 29%|██▊ | 203/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.linear_attn.conv1d.bias (f32, [8192]): 29%|██▊ | 204/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.linear_attn.dt_bias (f32, [32]): 29%|██▉ | 205/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.linear_attn.neg_exp_A (f32, [32]): 29%|██▉ | 206/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.linear_attn.norm.weight (f32, [128]): 29%|██▉ | 207/712 [00:01<00:08, 62.98it/s] Saving model.layers.25.post_attention_layernorm.weight (f32, [2560]): 29%|██▉ | 208/712 [00:01<00:08, 62.98it/s] Saving model.layers.26.input_layernorm.weight (f32, [2560]): 29%|██▉ | 209/712 [00:01<00:07, 62.98it/s] Saving model.layers.26.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 29%|██▉ | 210/712 [00:01<00:07, 62.98it/s] Saving model.layers.26.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 30%|██▉ | 211/712 [00:01<00:07, 62.98it/s] Saving model.layers.26.linear_attn.conv1d.weight (f32, [8192,1,4]): 30%|██▉ | 212/712 [00:01<00:07, 62.98it/s] Saving model.layers.26.linear_attn.conv1d.bias (f32, [8192]): 30%|██▉ | 213/712 [00:01<00:07, 62.98it/s] Saving model.layers.26.linear_attn.dt_bias (f32, [32]): 30%|███ | 214/712 [00:01<00:07, 62.98it/s] Saving model.layers.26.linear_attn.neg_exp_A (f32, [32]): 30%|███ | 215/712 [00:01<00:07, 62.98it/s] Saving model.layers.26.linear_attn.norm.weight (f32, [128]): 30%|███ | 216/712 [00:01<00:07, 62.98it/s] Saving model.layers.26.post_attention_layernorm.weight (f32, [2560]): 30%|███ | 217/712 [00:01<00:07, 62.98it/s] Saving model.layers.27.input_layernorm.weight (f32, [2560]): 31%|███ | 218/712 [00:01<00:07, 62.98it/s] Saving model.layers.27.attn.q_norm.layernorm.weight (f32, [256]): 31%|███ | 219/712 [00:01<00:07, 62.98it/s] Saving model.layers.27.attn.k_norm.layernorm.weight (f32, [256]): 31%|███ | 220/712 [00:01<00:07, 62.98it/s] Saving model.layers.27.post_attention_layernorm.weight (f32, [2560]): 31%|███ | 221/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.input_layernorm.weight (f32, [2560]): 31%|███ | 222/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 31%|███▏ | 223/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 31%|███▏ | 224/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.linear_attn.conv1d.weight (f32, [8192,1,4]): 32%|███▏ | 225/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.linear_attn.conv1d.bias (f32, [8192]): 32%|███▏ | 226/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.linear_attn.dt_bias (f32, [32]): 32%|███▏ | 227/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.linear_attn.neg_exp_A (f32, [32]): 32%|███▏ | 228/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.linear_attn.norm.weight (f32, [128]): 32%|███▏ | 229/712 [00:01<00:07, 62.98it/s] Saving model.layers.28.post_attention_layernorm.weight (f32, [2560]): 32%|███▏ | 230/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.input_layernorm.weight (f32, [2560]): 32%|███▏ | 231/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 33%|███▎ | 232/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 33%|███▎ | 233/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.linear_attn.conv1d.weight (f32, [8192,1,4]): 33%|███▎ | 234/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.linear_attn.conv1d.bias (f32, [8192]): 33%|███▎ | 235/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.linear_attn.dt_bias (f32, [32]): 33%|███▎ | 236/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.linear_attn.neg_exp_A (f32, [32]): 33%|███▎ | 237/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.linear_attn.norm.weight (f32, [128]): 33%|███▎ | 238/712 [00:01<00:07, 62.98it/s] Saving model.layers.29.post_attention_layernorm.weight (f32, [2560]): 34%|███▎ | 239/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.input_layernorm.weight (f32, [2560]): 34%|███▎ | 240/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 34%|███▍ | 241/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 34%|███▍ | 242/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.linear_attn.conv1d.weight (f32, [8192,1,4]): 34%|███▍ | 243/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.linear_attn.conv1d.bias (f32, [8192]): 34%|███▍ | 244/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.linear_attn.dt_bias (f32, [32]): 34%|███▍ | 245/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.linear_attn.neg_exp_A (f32, [32]): 35%|███▍ | 246/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.linear_attn.norm.weight (f32, [128]): 35%|███▍ | 247/712 [00:01<00:07, 62.98it/s] Saving model.layers.30.post_attention_layernorm.weight (f32, [2560]): 35%|███▍ | 248/712 [00:01<00:07, 62.98it/s] Saving model.layers.31.input_layernorm.weight (f32, [2560]): 35%|███▍ | 249/712 [00:01<00:07, 62.98it/s] Saving model.layers.31.attn.q_norm.layernorm.weight (f32, [256]): 35%|███▌ | 250/712 [00:01<00:07, 62.98it/s] Saving model.layers.31.attn.k_norm.layernorm.weight (f32, [256]): 35%|███▌ | 251/712 [00:01<00:07, 62.98it/s] Saving model.layers.31.post_attention_layernorm.weight (f32, [2560]): 35%|███▌ | 252/712 [00:01<00:07, 62.98it/s] Saving model.layers.32.final_norm_layernorm.weight (f32, [2560]): 36%|███▌ | 253/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 36%|███▌ | 254/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 36%|███▌ | 255/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 36%|███▌ | 256/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 36%|███▌ | 257/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 36%|███▌ | 258/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 36%|███▋ | 259/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 37%|███▋ | 260/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 37%|███▋ | 261/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 37%|███▋ | 262/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 37%|███▋ | 263/712 [00:01<00:07, 62.98it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 37%|███▋ | 264/712 [00:01<00:00, 461.46it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 37%|███▋ | 264/712 [00:01<00:00, 461.46it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 37%|███▋ | 265/712 [00:01<00:00, 461.46it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 37%|███▋ | 266/712 [00:01<00:00, 461.46it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 38%|███▊ | 267/712 [00:01<00:00, 461.46it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 38%|███▊ | 268/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 38%|███▊ | 269/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 38%|███▊ | 270/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 38%|███▊ | 271/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 38%|███▊ | 272/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 38%|███▊ | 273/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 38%|███▊ | 274/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 39%|███▊ | 275/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 39%|███▉ | 276/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 39%|███▉ | 277/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 39%|███▉ | 278/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 39%|███▉ | 279/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 39%|███▉ | 280/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 39%|███▉ | 281/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 40%|███▉ | 282/712 [00:01<00:00, 461.46it/s] Saving model.layers.1.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 40%|███▉ | 283/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 40%|███▉ | 284/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 40%|████ | 285/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 40%|████ | 286/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 40%|████ | 287/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 40%|████ | 288/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 41%|████ | 289/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 41%|████ | 290/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 41%|████ | 291/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 41%|████ | 292/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 41%|████ | 293/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 41%|████▏ | 294/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 41%|████▏ | 295/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 42%|████▏ | 296/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 42%|████▏ | 297/712 [00:01<00:00, 461.46it/s] Saving model.layers.2.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 42%|████▏ | 298/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 42%|████▏ | 299/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 42%|████▏ | 300/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 42%|████▏ | 301/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 42%|████▏ | 302/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 43%|████▎ | 303/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 43%|████▎ | 304/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 43%|████▎ | 305/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 43%|████▎ | 306/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 43%|████▎ | 307/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 43%|████▎ | 308/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 43%|████▎ | 309/712 [00:01<00:00, 461.46it/s] Saving model.layers.3.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 44%|████▎ | 310/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 44%|████▎ | 311/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 44%|████▍ | 312/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 44%|████▍ | 313/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 44%|████▍ | 314/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 44%|████▍ | 315/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 44%|████▍ | 316/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 45%|████▍ | 317/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 45%|████▍ | 318/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 45%|████▍ | 319/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 45%|████▍ | 320/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 45%|████▌ | 321/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 45%|████▌ | 322/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 45%|████▌ | 323/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 46%|████▌ | 324/712 [00:01<00:00, 461.46it/s] Saving model.layers.4.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 46%|████▌ | 325/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 46%|████▌ | 326/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 46%|████▌ | 327/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 46%|████▌ | 328/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 46%|████▌ | 329/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 46%|████▋ | 330/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 46%|████▋ | 331/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 47%|████▋ | 332/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 47%|████▋ | 333/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 47%|████▋ | 334/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 47%|████▋ | 335/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 47%|████▋ | 336/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 47%|████▋ | 337/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 47%|████▋ | 338/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 48%|████▊ | 339/712 [00:01<00:00, 461.46it/s] Saving model.layers.5.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 48%|████▊ | 340/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 48%|████▊ | 341/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 48%|████▊ | 342/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 48%|████▊ | 343/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 48%|████▊ | 344/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 48%|████▊ | 345/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 49%|████▊ | 346/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 49%|████▊ | 347/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 49%|████▉ | 348/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 49%|████▉ | 349/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 49%|████▉ | 350/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 49%|████▉ | 351/712 [00:01<00:00, 461.46it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 49%|████▉ | 352/712 [00:01<00:01, 229.40it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 49%|████▉ | 352/712 [00:01<00:01, 229.40it/s] Saving model.layers.6.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 50%|████▉ | 353/712 [00:01<00:01, 229.40it/s] Saving model.layers.6.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 50%|████▉ | 354/712 [00:01<00:01, 229.40it/s] Saving model.layers.6.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 50%|████▉ | 355/712 [00:01<00:01, 229.40it/s] Saving model.layers.7.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 50%|█████ | 356/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 50%|█████ | 357/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 50%|█████ | 358/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 50%|█████ | 359/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 51%|█████ | 360/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 51%|█████ | 361/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 51%|█████ | 362/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 51%|█████ | 363/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 51%|█████ | 364/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 51%|█████▏ | 365/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 51%|█████▏ | 366/712 [00:02<00:01, 229.40it/s] Saving model.layers.7.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 52%|█████▏ | 367/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 52%|█████▏ | 368/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 52%|█████▏ | 369/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 52%|█████▏ | 370/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 52%|█████▏ | 371/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 52%|█████▏ | 372/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 52%|█████▏ | 373/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 53%|█████▎ | 374/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 53%|█████▎ | 375/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 53%|█████▎ | 376/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 53%|█████▎ | 377/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 53%|█████▎ | 378/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 53%|█████▎ | 379/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 53%|█████▎ | 380/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 54%|█████▎ | 381/712 [00:02<00:01, 229.40it/s] Saving model.layers.8.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 54%|█████▎ | 382/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 54%|█████▍ | 383/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 54%|█████▍ | 384/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 54%|█████▍ | 385/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 54%|█████▍ | 386/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 54%|█████▍ | 387/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 54%|█████▍ | 388/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 55%|█████▍ | 389/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 55%|█████▍ | 390/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 55%|█████▍ | 391/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 55%|█████▌ | 392/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 55%|█████▌ | 393/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 55%|█████▌ | 394/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 55%|█████▌ | 395/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 56%|█████▌ | 396/712 [00:02<00:01, 229.40it/s] Saving model.layers.9.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 56%|█████▌ | 397/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 56%|█████▌ | 398/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 56%|█████▌ | 399/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 56%|█████▌ | 400/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 56%|█████▋ | 401/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 56%|█████▋ | 402/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 57%|█████▋ | 403/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 57%|█████▋ | 404/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 57%|█████▋ | 405/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 57%|█████▋ | 406/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 57%|█████▋ | 407/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 57%|█████▋ | 408/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 57%|█████▋ | 409/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 58%|█████▊ | 410/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 58%|█████▊ | 411/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 58%|█████▊ | 412/712 [00:02<00:01, 229.40it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 58%|█████▊ | 413/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 58%|█████▊ | 413/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 58%|█████▊ | 414/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 58%|█████▊ | 415/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 58%|█████▊ | 416/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 59%|█████▊ | 417/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 59%|█████▊ | 418/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 59%|█████▉ | 419/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 59%|█████▉ | 420/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 59%|█████▉ | 421/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 59%|█████▉ | 422/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 59%|█████▉ | 423/712 [00:02<00:01, 180.41it/s] Saving model.layers.11.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 60%|█████▉ | 424/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 60%|█████▉ | 425/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 60%|█████▉ | 426/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 60%|█████▉ | 427/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 60%|██████ | 428/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 60%|██████ | 429/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 60%|██████ | 430/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 61%|██████ | 431/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 61%|██████ | 432/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 61%|██████ | 433/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 61%|██████ | 434/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 61%|██████ | 435/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 61%|██████ | 436/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 61%|██████▏ | 437/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 62%|██████▏ | 438/712 [00:02<00:01, 180.41it/s] Saving model.layers.12.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 62%|██████▏ | 439/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 62%|██████▏ | 440/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 62%|██████▏ | 441/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 62%|██████▏ | 442/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 62%|██████▏ | 443/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 62%|██████▏ | 444/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 62%|██████▎ | 445/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 63%|██████▎ | 446/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 63%|██████▎ | 447/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 63%|██████▎ | 448/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 63%|██████▎ | 449/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 63%|██████▎ | 450/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 63%|██████▎ | 451/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 63%|██████▎ | 452/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 64%|██████▎ | 453/712 [00:02<00:01, 180.41it/s] Saving model.layers.13.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 64%|██████▍ | 454/712 [00:02<00:01, 180.41it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 64%|██████▍ | 455/712 [00:02<00:01, 180.41it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 64%|██████▍ | 456/712 [00:02<00:01, 180.41it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 64%|██████▍ | 457/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 64%|██████▍ | 457/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 64%|██████▍ | 458/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 64%|██████▍ | 459/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 65%|██████▍ | 460/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 65%|██████▍ | 461/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 65%|██████▍ | 462/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 65%|██████▌ | 463/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 65%|██████▌ | 464/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 65%|██████▌ | 465/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 65%|██████▌ | 466/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 66%|██████▌ | 467/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 66%|██████▌ | 468/712 [00:02<00:01, 164.09it/s] Saving model.layers.14.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 66%|██████▌ | 469/712 [00:02<00:01, 164.09it/s] Saving model.layers.15.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 66%|██████▌ | 470/712 [00:02<00:01, 164.09it/s] Saving model.layers.15.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 66%|██████▌ | 471/712 [00:02<00:01, 164.09it/s] Saving model.layers.15.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 66%|██████▋ | 472/712 [00:02<00:01, 164.09it/s] Saving model.layers.15.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 66%|██████▋ | 473/712 [00:03<00:01, 164.09it/s] Saving model.layers.15.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 67%|██████▋ | 474/712 [00:03<00:01, 164.09it/s] Saving model.layers.15.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 67%|██████▋ | 475/712 [00:03<00:01, 164.09it/s] Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 67%|██████▋ | 476/712 [00:03<00:01, 164.09it/s] Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 67%|██████▋ | 477/712 [00:03<00:01, 164.09it/s] Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 67%|██████▋ | 478/712 [00:03<00:01, 164.09it/s] Saving model.layers.15.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 67%|██████▋ | 479/712 [00:03<00:01, 164.09it/s] Saving model.layers.15.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 67%|██████▋ | 480/712 [00:03<00:01, 164.09it/s] Saving model.layers.15.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 68%|██████▊ | 481/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 68%|██████▊ | 482/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 68%|██████▊ | 483/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 68%|██████▊ | 484/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 68%|██████▊ | 485/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 68%|██████▊ | 486/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 68%|██████▊ | 487/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 69%|██████▊ | 488/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 69%|██████▊ | 489/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 69%|██████▉ | 490/712 [00:03<00:01, 164.09it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 69%|██████▉ | 491/712 [00:03<00:01, 155.22it/s] Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 69%|██████▉ | 491/712 [00:03<00:01, 155.22it/s] Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 69%|██████▉ | 492/712 [00:03<00:01, 155.22it/s] Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 69%|██████▉ | 493/712 [00:03<00:01, 155.22it/s] Saving model.layers.16.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 69%|██████▉ | 494/712 [00:03<00:01, 155.22it/s] Saving model.layers.16.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 70%|██████▉ | 495/712 [00:03<00:01, 155.22it/s] Saving model.layers.16.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 70%|██████▉ | 496/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 70%|██████▉ | 497/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 70%|██████▉ | 498/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 70%|███████ | 499/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 70%|███████ | 500/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 70%|███████ | 501/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 71%|███████ | 502/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 71%|███████ | 503/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 71%|███████ | 504/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 71%|███████ | 505/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 71%|███████ | 506/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 71%|███████ | 507/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 71%|███████▏ | 508/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 71%|███████▏ | 509/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 72%|███████▏ | 510/712 [00:03<00:01, 155.22it/s] Saving model.layers.17.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 72%|███████▏ | 511/712 [00:03<00:01, 155.22it/s] Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 72%|███████▏ | 512/712 [00:03<00:01, 155.22it/s] Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 72%|███████▏ | 513/712 [00:03<00:01, 155.22it/s] Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 72%|███████▏ | 514/712 [00:03<00:01, 155.22it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 72%|███████▏ | 515/712 [00:03<00:01, 155.22it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 72%|███████▏ | 516/712 [00:03<00:01, 155.22it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 73%|███████▎ | 517/712 [00:03<00:01, 155.22it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 73%|███████▎ | 518/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 73%|███████▎ | 518/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 73%|███████▎ | 519/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 73%|███████▎ | 520/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 73%|███████▎ | 521/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 73%|███████▎ | 522/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 73%|███████▎ | 523/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 74%|███████▎ | 524/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 74%|███████▎ | 525/712 [00:03<00:01, 150.24it/s] Saving model.layers.18.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 74%|███████▍ | 526/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 74%|███████▍ | 527/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 74%|███████▍ | 528/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 74%|███████▍ | 529/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 74%|███████▍ | 530/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 75%|███████▍ | 531/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 75%|███████▍ | 532/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 75%|███████▍ | 533/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 75%|███████▌ | 534/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 75%|███████▌ | 535/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 75%|███████▌ | 536/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 75%|███████▌ | 537/712 [00:03<00:01, 150.24it/s] Saving model.layers.19.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 76%|███████▌ | 538/712 [00:03<00:01, 150.24it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 76%|███████▌ | 539/712 [00:03<00:01, 150.24it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 76%|███████▌ | 540/712 [00:03<00:01, 150.24it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 76%|███████▌ | 541/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 76%|███████▌ | 541/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 76%|███████▌ | 542/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 76%|███████▋ | 543/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 76%|███████▋ | 544/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 77%|███████▋ | 545/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 77%|███████▋ | 546/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 77%|███████▋ | 547/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 77%|███████▋ | 548/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 77%|███████▋ | 549/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 77%|███████▋ | 550/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 77%|███████▋ | 551/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 78%|███████▊ | 552/712 [00:03<00:01, 142.44it/s] Saving model.layers.20.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 78%|███████▊ | 553/712 [00:03<00:01, 142.44it/s] Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 78%|███████▊ | 554/712 [00:03<00:01, 142.44it/s] Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 78%|███████▊ | 555/712 [00:03<00:01, 142.44it/s] Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 78%|███████▊ | 556/712 [00:03<00:01, 142.44it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 78%|███████▊ | 557/712 [00:03<00:01, 142.44it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 78%|███████▊ | 558/712 [00:03<00:01, 142.44it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 79%|███████▊ | 559/712 [00:03<00:01, 142.44it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 79%|███████▊ | 560/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 79%|███████▊ | 560/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 79%|███████▉ | 561/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 79%|███████▉ | 562/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 79%|███████▉ | 563/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 79%|███████▉ | 564/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 79%|███████▉ | 565/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 79%|███████▉ | 566/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 80%|███████▉ | 567/712 [00:03<00:01, 140.86it/s] Saving model.layers.21.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 80%|███████▉ | 568/712 [00:03<00:01, 140.86it/s] Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 80%|███████▉ | 569/712 [00:03<00:01, 140.86it/s] Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 80%|████████ | 570/712 [00:03<00:01, 140.86it/s] Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 80%|████████ | 571/712 [00:03<00:01, 140.86it/s] Saving model.layers.22.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 80%|████████ | 572/712 [00:03<00:00, 140.86it/s] Saving model.layers.22.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 80%|████████ | 573/712 [00:03<00:00, 140.86it/s] Saving model.layers.22.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 81%|████████ | 574/712 [00:03<00:00, 140.86it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 81%|████████ | 575/712 [00:03<00:00, 140.86it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 81%|████████ | 576/712 [00:03<00:00, 140.86it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 81%|████████ | 577/712 [00:03<00:00, 140.86it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 81%|████████ | 578/712 [00:03<00:00, 142.43it/s] Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 81%|████████ | 578/712 [00:03<00:00, 142.43it/s] Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 81%|████████▏ | 579/712 [00:03<00:00, 142.43it/s] Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 81%|████████▏ | 580/712 [00:03<00:00, 142.43it/s] Saving model.layers.22.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 82%|████████▏ | 581/712 [00:03<00:00, 142.43it/s] Saving model.layers.22.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 82%|████████▏ | 582/712 [00:03<00:00, 142.43it/s] Saving model.layers.22.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 82%|████████▏ | 583/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 82%|████████▏ | 584/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 82%|████████▏ | 585/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 82%|████████▏ | 586/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 82%|████████▏ | 587/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 83%|████████▎ | 588/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 83%|████████▎ | 589/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 83%|████████▎ | 590/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 83%|████████▎ | 591/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 83%|████████▎ | 592/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 83%|████████▎ | 593/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 83%|████████▎ | 594/712 [00:03<00:00, 142.43it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 84%|████████▎ | 595/712 [00:03<00:00, 132.38it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 84%|████████▎ | 595/712 [00:03<00:00, 132.38it/s] Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 84%|████████▎ | 596/712 [00:03<00:00, 132.38it/s] Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 84%|████████▍ | 597/712 [00:03<00:00, 132.38it/s] Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 84%|████████▍ | 598/712 [00:03<00:00, 132.38it/s] Saving model.layers.24.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 84%|████████▍ | 599/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 84%|████████▍ | 600/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 84%|████████▍ | 601/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 85%|████████▍ | 602/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 85%|████████▍ | 603/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 85%|████████▍ | 604/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 85%|████████▍ | 605/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 85%|████████▌ | 606/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 85%|████████▌ | 607/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 85%|████████▌ | 608/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 86%|████████▌ | 609/712 [00:04<00:00, 132.38it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 86%|████████▌ | 610/712 [00:04<00:00, 131.94it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 86%|████████▌ | 610/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 86%|████████▌ | 611/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 86%|████████▌ | 612/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 86%|████████▌ | 613/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 86%|████████▌ | 614/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 86%|████████▋ | 615/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 87%|████████▋ | 616/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 87%|████████▋ | 617/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 87%|████████▋ | 618/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 87%|████████▋ | 619/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 87%|████████▋ | 620/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 87%|████████▋ | 621/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 87%|████████▋ | 622/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 88%|████████▊ | 623/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 88%|████████▊ | 624/712 [00:04<00:00, 131.94it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 88%|████████▊ | 625/712 [00:04<00:00, 131.68it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 88%|████████▊ | 625/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 88%|████████▊ | 626/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 88%|████████▊ | 627/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 88%|████████▊ | 628/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 88%|████████▊ | 629/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 88%|████████▊ | 630/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 89%|████████▊ | 631/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 89%|████████▉ | 632/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 89%|████████▉ | 633/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 89%|████████▉ | 634/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 89%|████████▉ | 635/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 89%|████████▉ | 636/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 89%|████████▉ | 637/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 90%|████████▉ | 638/712 [00:04<00:00, 131.68it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 90%|████████▉ | 639/712 [00:04<00:00, 130.27it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 90%|████████▉ | 639/712 [00:04<00:00, 130.27it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 90%|████████▉ | 640/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 90%|█████████ | 641/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 90%|█████████ | 642/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 90%|█████████ | 643/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 90%|█████████ | 644/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 91%|█████████ | 645/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 91%|█████████ | 646/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 91%|█████████ | 647/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 91%|█████████ | 648/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 91%|█████████ | 649/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 91%|█████████▏| 650/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 91%|█████████▏| 651/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 92%|█████████▏| 652/712 [00:04<00:00, 130.27it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 92%|█████████▏| 653/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 92%|█████████▏| 653/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 92%|█████████▏| 654/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 92%|█████████▏| 655/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 92%|█████████▏| 656/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 92%|█████████▏| 657/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 92%|█████████▏| 658/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 93%|█████████▎| 659/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 93%|█████████▎| 660/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 93%|█████████▎| 661/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 93%|█████████▎| 662/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 93%|█████████▎| 663/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 93%|█████████▎| 664/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 93%|█████████▎| 665/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 94%|█████████▎| 666/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 94%|█████████▎| 667/712 [00:04<00:00, 126.35it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 94%|█████████▍| 668/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 94%|█████████▍| 668/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 94%|█████████▍| 669/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 94%|█████████▍| 670/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 94%|█████████▍| 671/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 94%|█████████▍| 672/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 95%|█████████▍| 673/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 95%|█████████▍| 674/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 95%|█████████▍| 675/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 95%|█████████▍| 676/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 95%|█████████▌| 677/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 95%|█████████▌| 678/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 95%|█████████▌| 679/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 96%|█████████▌| 680/712 [00:04<00:00, 126.91it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 96%|█████████▌| 681/712 [00:04<00:00, 127.28it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 96%|█████████▌| 681/712 [00:04<00:00, 127.28it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 96%|█████████▌| 682/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,80]): 96%|█████████▌| 683/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 96%|█████████▌| 684/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_Q8G32 (u8, [8192,80,32]): 96%|█████████▌| 685/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,80]): 96%|█████████▋| 686/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 96%|█████████▋| 687/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.z_proj.MatMul.weight_Q8G32 (u8, [4096,80,32]): 97%|█████████▋| 688/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,128]): 97%|█████████▋| 689/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 97%|█████████▋| 690/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.linear_attn.out_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 97%|█████████▋| 691/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 97%|█████████▋| 692/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 97%|█████████▋| 693/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 97%|█████████▋| 694/712 [00:04<00:00, 127.28it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 98%|█████████▊| 695/712 [00:04<00:00, 122.97it/s] Saving model.layers.30.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 98%|█████████▊| 695/712 [00:04<00:00, 122.97it/s] Saving model.layers.30.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 98%|█████████▊| 696/712 [00:04<00:00, 122.97it/s] Saving model.layers.30.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 98%|█████████▊| 697/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 98%|█████████▊| 698/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 98%|█████████▊| 699/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 98%|█████████▊| 700/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 98%|█████████▊| 701/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 99%|█████████▊| 702/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 99%|█████████▊| 703/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 99%|█████████▉| 704/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 99%|█████████▉| 705/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 99%|█████████▉| 706/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 99%|█████████▉| 707/712 [00:04<00:00, 122.97it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 99%|█████████▉| 708/712 [00:04<00:00, 118.31it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 99%|█████████▉| 708/712 [00:04<00:00, 118.31it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 100%|█████████▉| 709/712 [00:04<00:00, 118.31it/s] Saving lm_head.MatMul.weight_scale (f32, [248320,80]): 100%|█████████▉| 710/712 [00:04<00:00, 118.31it/s] Saving lm_head.MatMul.weight_zp (u8, [248320,80]): 100%|█████████▉| 711/712 [00:04<00:00, 118.31it/s] Saving lm_head.MatMul.weight_Q8G32 (u8, [248320,80,32]): 100%|██████████| 712/712 [00:04<00:00, 118.31it/s] Saving lm_head.MatMul.weight_Q8G32 (u8, [248320,80,32]): 100%|██████████| 712/712 [00:05<00:00, 134.70it/s]
2026-11-06 13:06:23,455 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize model with k_quant algorithm...
2026-11-06 13:06:23,456 neural_compressor [WARNING] - Model size > 2GB. Please use model path instead of onnx model object to quantize
2026-11-06 13:06:23,477 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:06:25,745 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:06:30,118 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:06:31,898 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:06:39,484 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:06:43,976 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 13:08:40,673 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of model with k_quant algorithm.
Saving ONNX model in model-cpu-int8
Progress: [### ] 16.67% Progress: [###### ] 33.33% Progress: [########## ] 50.00% Progress: [############# ] 66.67% Progress: [################ ] 83.33% Progress: [####################] 100.00% 0it [00:00, ?it/s] Saving model.embed_tokens.weight_Q8 (u8, [248320,2560]): 3%|▎ | 1/29 [00:00<00:00, 95.76it/s] Saving model.embed_tokens.weight_Q8 (u8, [248320,2560]): 7%|▋ | 2/29 [00:00<00:04, 6.30it/s] Saving model.embed_tokens.weight_scales (f32, [248320,80]): 7%|▋ | 2/29 [00:00<00:04, 6.30it/s] Saving model.mtp.pre_fc_norm_embedding.weight (f32, [2560]): 10%|█ | 3/29 [00:00<00:04, 6.30it/s] Saving model.mtp.pre_fc_norm_hidden.weight (f32, [2560]): 14%|█▍ | 4/29 [00:00<00:03, 6.30it/s] Saving model.layers.0.input_layernorm.weight (f32, [2560]): 17%|█▋ | 5/29 [00:00<00:03, 6.30it/s] Saving model.layers.0.attn.q_norm.layernorm.weight (f32, [256]): 21%|██ | 6/29 [00:00<00:03, 6.30it/s] Saving model.layers.0.attn.k_norm.layernorm.weight (f32, [256]): 24%|██▍ | 7/29 [00:00<00:03, 6.30it/s] Saving cos_cache (f32, [262144,32]): 28%|██▊ | 8/29 [00:00<00:03, 6.30it/s] Saving sin_cache (f32, [262144,32]): 31%|███ | 9/29 [00:00<00:03, 6.30it/s] Saving model.layers.0.post_attention_layernorm.weight (f32, [2560]): 34%|███▍ | 10/29 [00:00<00:03, 6.30it/s] Saving model.layers.1.final_norm_layernorm.weight (f32, [2560]): 38%|███▊ | 11/29 [00:00<00:02, 6.30it/s] Saving model.mtp.fc.MatMul.weight_scale (f32, [2560,160]): 41%|████▏ | 12/29 [00:00<00:02, 6.30it/s] Saving model.mtp.fc.MatMul.weight_zp (u8, [2560,160]): 45%|████▍ | 13/29 [00:00<00:02, 6.30it/s] Saving model.mtp.fc.MatMul.weight_Q8G32 (u8, [2560,160,32]): 48%|████▊ | 14/29 [00:00<00:02, 6.30it/s] Saving model.layers.0.attn.qkv_proj.MatMul.weight_scale (f32, [10240,80]): 52%|█████▏ | 15/29 [00:00<00:02, 6.30it/s] Saving model.layers.0.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 55%|█████▌ | 16/29 [00:00<00:02, 6.30it/s] Saving model.layers.0.attn.qkv_proj.MatMul.weight_Q8G32 (u8, [10240,80,32]): 59%|█████▊ | 17/29 [00:00<00:01, 6.30it/s] Saving model.layers.0.attn.o_proj.MatMul.weight_scale (f32, [2560,128]): 62%|██████▏ | 18/29 [00:00<00:01, 6.30it/s] Saving model.layers.0.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 66%|██████▌ | 19/29 [00:00<00:01, 6.30it/s] Saving model.layers.0.attn.o_proj.MatMul.weight_Q8G32 (u8, [2560,128,32]): 69%|██████▉ | 20/29 [00:00<00:01, 6.30it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,80]): 72%|███████▏ | 21/29 [00:00<00:01, 6.30it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 76%|███████▌ | 22/29 [00:00<00:01, 6.30it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_Q8G32 (u8, [18432,80,32]): 79%|███████▉ | 23/29 [00:00<00:00, 6.30it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_scale (f32, [2560,288]): 83%|████████▎ | 24/29 [00:00<00:00, 6.30it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 86%|████████▌ | 25/29 [00:00<00:00, 6.30it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_Q8G32 (u8, [2560,288,32]): 90%|████████▉ | 26/29 [00:00<00:00, 6.30it/s] Saving lm_head.MatMul.weight_scale (f32, [248320,80]): 93%|█████████▎| 27/29 [00:00<00:00, 6.30it/s] Saving lm_head.MatMul.weight_scale (f32, [248320,80]): 97%|█████████▋| 28/29 [00:00<00:00, 81.98it/s] Saving lm_head.MatMul.weight_zp (u8, [248320,80]): 97%|█████████▋| 28/29 [00:00<00:00, 81.98it/s] Saving lm_head.MatMul.weight_Q8G32 (u8, [248320,80,32]): 100%|██████████| 29/29 [00:00<00:00, 81.98it/s] Saving lm_head.MatMul.weight_Q8G32 (u8, [248320,80,32]): 100%|██████████| 29/29 [00:00<00:00, 39.46it/s]
Configured 1450 MB of runtime initializer sharing; model-cpu-int8/mtp.onnx.data remains self-contained.
Warning: could not read generation_config.json (/home/anon/LLM/KnowLine-4B-Gen5.5 does not appear to have a file named generation_config.json. Checkout 'https://huggingface.co//home/anon/LLM/KnowLine-4B-Gen5.5/tree/main' for available files.). Falling back to config.json.
Adding the tokenizer's end-of-turn token <|im_end|> (id 248046) to eos_token_id from the model configuration: 248044.
Saving GenAI config in model-cpu-int8
Added 'mtp' section to genai_config.json
Saving processing files in model-cpu-int8 for GenAI
EXIT:0
Fixing tokenizer in model-cpu-int8
fixed model-cpu-int8/tokenizer.json
fixed model-cpu-int8/tokenizer_config.json
Done.