Hugging Face's logo

electroglyph
/
KnowLine-4B-Gen5.5-onnx

Text Classification
ONNX
onnxruntime
decision-model
qwen3.5
Model card Files
xet
 Community
KnowLine-4B-Gen5.5-onnx / export-cpu-new.log
electroglyph's picture
electroglyph
Upload folder using huggingface_hub
7d5fc54 verified
Raw History Blame Contribute Delete
154 kB
Valid precision + execution provider combinations are: FP32 CPU, FP32 CUDA, FP16 CUDA, FP16 DML, FP16 TRT-RTX, BF16 CUDA, BF16 TRT-RTX, INT8 CPU, INT8 CUDA, INT8 WebGPU, INT4 CPU, INT4 CUDA, INT4 DML, INT4 WebGPU
Extra options: {'accuracy_level': '2', 'block_size': '16', 'algo_config': 'k_quant'}
GroupQueryAttention (GQA) is used in this model.
GroupQueryAttention (GQA) is used in this model.
Loading weights: 0%| | 0/426 [00:00<?, ?it/s] Loading weights: 100%|██████████| 426/426 [00:00<00:00, 23609.59it/s]
2026-11-06 00:55:37,611 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize model with k_quant algorithm...
2026-11-06 00:55:37,621 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_Q4G16' of node '/model/embed_tokens/Reshape' (::Reshape:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-06 00:55:37,621 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_scale' of node '/model/embed_tokens/GatherBlockQuantized' (com.microsoft::GatherBlockQuantized:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-06 00:55:37,621 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_zp' of node '/model/embed_tokens/GatherBlockQuantized' (com.microsoft::GatherBlockQuantized:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-06 00:55:37,630 neural_compressor [WARNING] - Model size > 2GB. Please use model path instead of onnx model object to quantize
2026-11-06 00:55:37,654 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:55:41,701 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:55:43,644 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:55:45,611 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:55:54,028 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:55:58,218 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:02,214 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:04,090 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:05,970 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:14,311 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:18,481 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:22,419 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:24,269 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:26,122 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:34,390 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:38,543 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:43,179 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:45,042 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:53,324 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:56:57,471 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:01,447 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:03,308 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:05,206 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:13,543 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:17,701 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:21,594 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:23,436 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:25,351 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:33,626 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:37,777 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:41,743 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:43,640 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:45,567 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:53,864 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:57:58,036 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:02,687 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:04,538 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:12,831 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:16,994 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:20,951 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:22,868 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:24,746 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:33,065 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:37,235 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:41,199 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:43,035 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:44,966 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:53,182 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:58:57,356 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:01,312 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:03,247 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:05,158 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:13,437 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:17,590 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:22,247 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:24,130 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:32,450 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:36,654 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:40,540 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:42,430 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:44,347 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:52,654 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 00:59:56,817 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:00,732 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:02,583 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:04,452 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:12,793 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:16,940 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:20,859 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:22,717 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:24,608 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:32,916 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:37,075 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:41,699 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:43,580 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:51,885 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:56,034 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:00:59,991 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:01,929 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:03,790 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:12,046 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:16,213 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:20,204 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:22,051 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:23,905 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:32,198 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:36,379 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:40,337 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:42,174 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:44,050 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:52,372 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:01:56,527 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:01,160 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:03,109 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:11,393 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:15,581 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:19,595 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:21,515 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:23,438 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:31,695 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:35,817 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:39,798 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:41,700 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:43,564 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:51,805 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:55,940 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:02:59,833 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:01,743 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:03,684 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:11,971 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:16,147 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:20,797 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:22,658 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:30,970 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:35,114 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:39,068 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:40,994 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:42,940 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:51,258 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:55,411 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:03:59,377 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:01,223 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:03,086 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:11,380 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:15,528 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:19,461 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:21,303 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:23,249 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:31,553 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:35,717 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:40,354 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:42,210 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:50,494 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:54,667 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:04:58,615 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:00,459 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:02,331 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:10,644 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:14,809 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:18,713 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:20,629 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:22,490 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
Reading embedding layer
Reading layer 0
Reading layer 1
Reading layer 2
Reading layer 3
Reading layer 4
Reading layer 5
Reading layer 6
Reading layer 7
Reading layer 8
Reading layer 9
Reading layer 10
Reading layer 11
Reading layer 12
Reading layer 13
Reading layer 14
Reading layer 15
Reading layer 16
Reading layer 17
Reading layer 18
Reading layer 19
Reading layer 20
Reading layer 21
Reading layer 22
Reading layer 23
Reading layer 24
Reading layer 25
Reading layer 26
Reading layer 27
Reading layer 28
Reading layer 29
Reading layer 30
Reading layer 31
Reading final norm
Reading LM head
Building MTP (multi-token prediction) head -> mtp.onnx
Saving ONNX model in model-cpu-new
Progress: [ ] 0.50% Progress: [ ] 1.00% Progress: [ ] 1.49% Progress: [ ] 1.99% Progress: [ ] 2.49% Progress: [ ] 2.99% Progress: [ ] 3.48% Progress: [ ] 3.98% Progress: [ ] 4.48% Progress: [ ] 4.98% Progress: [# ] 5.47% Progress: [# ] 5.97% Progress: [# ] 6.47% Progress: [# ] 6.97% Progress: [# ] 7.46% Progress: [# ] 7.96% Progress: [# ] 8.46% Progress: [# ] 8.96% Progress: [# ] 9.45% Progress: [# ] 9.95% Progress: [## ] 10.45% Progress: [## ] 10.95% Progress: [## ] 11.44% Progress: [## ] 11.94% Progress: [## ] 12.44% Progress: [## ] 12.94% Progress: [## ] 13.43% Progress: [## ] 13.93% Progress: [## ] 14.43% Progress: [## ] 14.93% Progress: [### ] 15.42% Progress: [### ] 15.92% Progress: [### ] 16.42% Progress: [### ] 16.92% Progress: [### ] 17.41% Progress: [### ] 17.91% Progress: [### ] 18.41% Progress: [### ] 18.91% Progress: [### ] 19.40% Progress: [### ] 19.90% Progress: [#### ] 20.40% Progress: [#### ] 20.90% Progress: [#### ] 21.39% Progress: [#### ] 21.89% Progress: [#### ] 22.39% Progress: [#### ] 22.89% Progress: [#### ] 23.38% Progress: [#### ] 23.88% Progress: [#### ] 24.38% Progress: [#### ] 24.88% Progress: [##### ] 25.37% Progress: [##### ] 25.87% Progress: [##### ] 26.37% Progress: [##### ] 26.87% Progress: [##### ] 27.36% Progress: [##### ] 27.86% Progress: [##### ] 28.36% Progress: [##### ] 28.86% Progress: [##### ] 29.35% Progress: [##### ] 29.85% Progress: [###### ] 30.35% Progress: [###### ] 30.85% Progress: [###### ] 31.34% Progress: [###### ] 31.84% Progress: [###### ] 32.34% Progress: [###### ] 32.84% Progress: [###### ] 33.33% Progress: [###### ] 33.83% Progress: [###### ] 34.33% Progress: [###### ] 34.83% Progress: [####### ] 35.32% Progress: [####### ] 35.82% Progress: [####### ] 36.32% Progress: [####### ] 36.82% Progress: [####### ] 37.31% Progress: [####### ] 37.81% Progress: [####### ] 38.31% Progress: [####### ] 38.81% Progress: [####### ] 39.30% Progress: [####### ] 39.80% Progress: [######## ] 40.30% Progress: [######## ] 40.80% Progress: [######## ] 41.29% Progress: [######## ] 41.79% Progress: [######## ] 42.29% Progress: [######## ] 42.79% Progress: [######## ] 43.28% Progress: [######## ] 43.78% Progress: [######## ] 44.28% Progress: [######## ] 44.78% Progress: [######### ] 45.27% Progress: [######### ] 45.77% Progress: [######### ] 46.27% Progress: [######### ] 46.77% Progress: [######### ] 47.26% Progress: [######### ] 47.76% Progress: [######### ] 48.26% Progress: [######### ] 48.76% Progress: [######### ] 49.25% Progress: [######### ] 49.75% Progress: [########## ] 50.25% Progress: [########## ] 50.75% Progress: [########## ] 51.24% Progress: [########## ] 51.74% Progress: [########## ] 52.24% Progress: [########## ] 52.74% Progress: [########## ] 53.23% Progress: [########## ] 53.73% Progress: [########## ] 54.23% Progress: [########## ] 54.73% Progress: [########### ] 55.22% Progress: [########### ] 55.72% Progress: [########### ] 56.22% Progress: [########### ] 56.72% Progress: [########### ] 57.21% Progress: [########### ] 57.71% Progress: [########### ] 58.21% Progress: [########### ] 58.71% Progress: [########### ] 59.20% Progress: [########### ] 59.70% Progress: [############ ] 60.20% Progress: [############ ] 60.70% Progress: [############ ] 61.19% Progress: [############ ] 61.69% Progress: [############ ] 62.19% Progress: [############ ] 62.69% Progress: [############ ] 63.18% Progress: [############ ] 63.68% Progress: [############ ] 64.18% Progress: [############ ] 64.68% Progress: [############# ] 65.17% Progress: [############# ] 65.67% Progress: [############# ] 66.17% Progress: [############# ] 66.67% Progress: [############# ] 67.16% Progress: [############# ] 67.66% Progress: [############# ] 68.16% Progress: [############# ] 68.66% Progress: [############# ] 69.15% Progress: [############# ] 69.65% Progress: [############## ] 70.15% Progress: [############## ] 70.65% Progress: [############## ] 71.14% Progress: [############## ] 71.64% Progress: [############## ] 72.14% Progress: [############## ] 72.64% Progress: [############## ] 73.13% Progress: [############## ] 73.63% Progress: [############## ] 74.13% Progress: [############## ] 74.63% Progress: [############### ] 75.12% Progress: [############### ] 75.62% Progress: [############### ] 76.12% Progress: [############### ] 76.62% Progress: [############### ] 77.11% Progress: [############### ] 77.61% Progress: [############### ] 78.11% Progress: [############### ] 78.61% Progress: [############### ] 79.10% Progress: [############### ] 79.60% Progress: [################ ] 80.10% Progress: [################ ] 80.60% Progress: [################ ] 81.09% Progress: [################ ] 81.59% Progress: [################ ] 82.09% Progress: [################ ] 82.59% Progress: [################ ] 83.08% Progress: [################ ] 83.58% Progress: [################ ] 84.08% Progress: [################ ] 84.58% Progress: [################# ] 85.07% Progress: [################# ] 85.57% Progress: [################# ] 86.07% Progress: [################# ] 86.57% Progress: [################# ] 87.06% Progress: [################# ] 87.56% Progress: [################# ] 88.06% Progress: [################# ] 88.56% Progress: [################# ] 89.05% Progress: [################# ] 89.55% Progress: [################## ] 90.05% Progress: [################## ] 90.55% Progress: [################## ] 91.04% Progress: [################## ] 91.54% Progress: [################## ] 92.04% Progress: [################## ] 92.54% Progress: [################## ] 93.03% Progress: [################## ] 93.53% Progress: [################## ] 94.03%2026-11-06 01:05:30,812 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:34,967 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:38,948 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:40,813 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:42,677 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:50,966 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:55,116 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:05:59,744 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:06:01,602 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:06:09,910 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:06:14,629 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:10:13,408 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of model with k_quant algorithm.
Progress: [################## ] 94.53% Progress: [################### ] 95.02% Progress: [################### ] 95.52% Progress: [################### ] 96.02% Progress: [################### ] 96.52% Progress: [################### ] 97.01% Progress: [################### ] 97.51% Progress: [################### ] 98.01% Progress: [################### ] 98.51% Progress: [################### ] 99.00% Progress: [################### ] 99.50% Progress: [####################] 100.00% 0it [00:00, ?it/s] Saving model.layers.0.input_layernorm.weight (f32, [2560]): 0%| | 1/710 [00:00<00:05, 120.58it/s] Saving model.layers.0.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 0%| | 2/710 [00:00<00:02, 238.60it/s] Saving model.layers.0.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 0%| | 3/710 [00:00<00:02, 353.25it/s] Saving model.layers.0.linear_attn.conv1d.weight (f32, [8192,1,4]): 1%| | 4/710 [00:00<00:01, 466.19it/s] Saving model.layers.0.linear_attn.conv1d.bias (f32, [8192]): 1%| | 5/710 [00:00<00:01, 577.87it/s] Saving model.layers.0.linear_attn.dt_bias (f32, [32]): 1%| | 6/710 [00:00<00:01, 690.10it/s] Saving model.layers.0.linear_attn.neg_exp_A (f32, [32]): 1%| | 7/710 [00:00<00:00, 802.78it/s] Saving model.layers.0.linear_attn.norm.weight (f32, [128]): 1%| | 8/710 [00:00<00:00, 914.76it/s] Saving model.layers.0.post_attention_layernorm.weight (f32, [2560]): 1%|▏ | 9/710 [00:00<00:00, 1026.25it/s] Saving model.layers.1.input_layernorm.weight (f32, [2560]): 1%|▏ | 10/710 [00:00<00:00, 1135.28it/s] Saving model.layers.1.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 2%|▏ | 11/710 [00:00<00:00, 1243.96it/s] Saving model.layers.1.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 2%|▏ | 12/710 [00:00<00:00, 1345.37it/s] Saving model.layers.1.linear_attn.conv1d.weight (f32, [8192,1,4]): 2%|▏ | 13/710 [00:00<00:00, 1445.01it/s] Saving model.layers.1.linear_attn.conv1d.bias (f32, [8192]): 2%|▏ | 14/710 [00:00<00:00, 1546.65it/s] Saving model.layers.1.linear_attn.dt_bias (f32, [32]): 2%|▏ | 15/710 [00:00<00:00, 1649.05it/s] Saving model.layers.1.linear_attn.neg_exp_A (f32, [32]): 2%|▏ | 16/710 [00:00<00:00, 1754.48it/s] Saving model.layers.1.linear_attn.norm.weight (f32, [128]): 2%|▏ | 17/710 [00:00<00:00, 1858.55it/s] Saving model.layers.1.post_attention_layernorm.weight (f32, [2560]): 3%|▎ | 18/710 [00:00<00:00, 1962.71it/s] Saving model.layers.2.input_layernorm.weight (f32, [2560]): 3%|▎ | 19/710 [00:00<00:00, 2065.09it/s] Saving model.layers.2.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 3%|▎ | 20/710 [00:00<00:00, 2167.26it/s] Saving model.layers.2.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 3%|▎ | 21/710 [00:00<00:00, 2258.30it/s] Saving model.layers.2.linear_attn.conv1d.weight (f32, [8192,1,4]): 3%|▎ | 22/710 [00:00<00:00, 2345.75it/s] Saving model.layers.2.linear_attn.conv1d.bias (f32, [8192]): 3%|▎ | 23/710 [00:00<00:00, 2438.61it/s] Saving model.layers.2.linear_attn.dt_bias (f32, [32]): 3%|▎ | 24/710 [00:00<00:00, 2534.83it/s] Saving model.layers.2.linear_attn.neg_exp_A (f32, [32]): 4%|▎ | 25/710 [00:00<00:00, 2634.35it/s] Saving model.layers.2.linear_attn.norm.weight (f32, [128]): 4%|▎ | 26/710 [00:00<00:00, 2732.86it/s] Saving model.layers.2.post_attention_layernorm.weight (f32, [2560]): 4%|▍ | 27/710 [00:00<00:00, 2830.80it/s] Saving model.layers.3.input_layernorm.weight (f32, [2560]): 4%|▍ | 28/710 [00:00<00:00, 2925.70it/s] Saving model.layers.3.attn.q_norm.layernorm.weight (f32, [256]): 4%|▍ | 29/710 [00:00<00:00, 3021.08it/s] Saving model.layers.3.attn.k_norm.layernorm.weight (f32, [256]): 4%|▍ | 30/710 [00:00<00:00, 3118.29it/s] Saving cos_cache (f32, [262144,32]): 4%|▍ | 31/710 [00:00<00:00, 3214.03it/s] Saving sin_cache (f32, [262144,32]): 5%|▍ | 32/710 [00:00<00:00, 2310.12it/s] Saving model.layers.3.post_attention_layernorm.weight (f32, [2560]): 5%|▍ | 33/710 [00:00<00:00, 1799.45it/s] Saving model.layers.4.input_layernorm.weight (f32, [2560]): 5%|▍ | 34/710 [00:00<00:00, 1843.63it/s] Saving model.layers.4.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 5%|▍ | 35/710 [00:00<00:00, 1894.28it/s] Saving model.layers.4.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 5%|▌ | 36/710 [00:00<00:00, 1938.87it/s] Saving model.layers.4.linear_attn.conv1d.weight (f32, [8192,1,4]): 5%|▌ | 37/710 [00:00<00:00, 1983.20it/s] Saving model.layers.4.linear_attn.conv1d.bias (f32, [8192]): 5%|▌ | 38/710 [00:00<00:00, 2030.44it/s] Saving model.layers.4.linear_attn.dt_bias (f32, [32]): 5%|▌ | 39/710 [00:00<00:00, 2079.66it/s] Saving model.layers.4.linear_attn.neg_exp_A (f32, [32]): 6%|▌ | 40/710 [00:00<00:00, 2130.47it/s] Saving model.layers.4.linear_attn.norm.weight (f32, [128]): 6%|▌ | 41/710 [00:00<00:00, 2180.93it/s] Saving model.layers.4.post_attention_layernorm.weight (f32, [2560]): 6%|▌ | 42/710 [00:00<00:00, 2231.49it/s] Saving model.layers.5.input_layernorm.weight (f32, [2560]): 6%|▌ | 43/710 [00:00<00:00, 2280.98it/s] Saving model.layers.5.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 6%|▌ | 44/710 [00:00<00:00, 2330.58it/s] Saving model.layers.5.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 6%|▋ | 45/710 [00:00<00:00, 2373.99it/s] Saving model.layers.5.linear_attn.conv1d.weight (f32, [8192,1,4]): 6%|▋ | 46/710 [00:00<00:00, 2416.23it/s] Saving model.layers.5.linear_attn.conv1d.bias (f32, [8192]): 7%|▋ | 47/710 [00:00<00:00, 2461.35it/s] Saving model.layers.5.linear_attn.dt_bias (f32, [32]): 7%|▋ | 48/710 [00:00<00:00, 2508.77it/s] Saving model.layers.5.linear_attn.neg_exp_A (f32, [32]): 7%|▋ | 49/710 [00:00<00:00, 2558.01it/s] Saving model.layers.5.linear_attn.norm.weight (f32, [128]): 7%|▋ | 50/710 [00:00<00:00, 2606.87it/s] Saving model.layers.5.post_attention_layernorm.weight (f32, [2560]): 7%|▋ | 51/710 [00:00<00:00, 2655.81it/s] Saving model.layers.6.input_layernorm.weight (f32, [2560]): 7%|▋ | 52/710 [00:00<00:00, 2703.52it/s] Saving model.layers.6.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 7%|▋ | 53/710 [00:00<00:00, 2751.49it/s] Saving model.layers.6.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 8%|▊ | 54/710 [00:00<00:00, 2792.24it/s] Saving model.layers.6.linear_attn.conv1d.weight (f32, [8192,1,4]): 8%|▊ | 55/710 [00:00<00:00, 2831.90it/s] Saving model.layers.6.linear_attn.conv1d.bias (f32, [8192]): 8%|▊ | 56/710 [00:00<00:00, 2873.13it/s] Saving model.layers.6.linear_attn.dt_bias (f32, [32]): 8%|▊ | 57/710 [00:00<00:00, 2918.01it/s] Saving model.layers.6.linear_attn.neg_exp_A (f32, [32]): 8%|▊ | 58/710 [00:00<00:00, 2965.65it/s] Saving model.layers.6.linear_attn.norm.weight (f32, [128]): 8%|▊ | 59/710 [00:00<00:00, 3012.93it/s] Saving model.layers.6.post_attention_layernorm.weight (f32, [2560]): 8%|▊ | 60/710 [00:00<00:00, 3060.46it/s] Saving model.layers.7.input_layernorm.weight (f32, [2560]): 9%|▊ | 61/710 [00:00<00:00, 3106.44it/s] Saving model.layers.7.attn.q_norm.layernorm.weight (f32, [256]): 9%|▊ | 62/710 [00:00<00:00, 3152.62it/s] Saving model.layers.7.attn.k_norm.layernorm.weight (f32, [256]): 9%|▉ | 63/710 [00:00<00:00, 3200.05it/s] Saving model.layers.7.post_attention_layernorm.weight (f32, [2560]): 9%|▉ | 64/710 [00:00<00:00, 3246.64it/s] Saving model.layers.8.input_layernorm.weight (f32, [2560]): 9%|▉ | 65/710 [00:00<00:00, 3291.87it/s] Saving model.layers.8.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 9%|▉ | 66/710 [00:00<00:00, 3337.64it/s] Saving model.layers.8.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 9%|▉ | 67/710 [00:00<00:00, 3373.97it/s] Saving model.layers.8.linear_attn.conv1d.weight (f32, [8192,1,4]): 10%|▉ | 68/710 [00:00<00:00, 3409.55it/s] Saving model.layers.8.linear_attn.conv1d.bias (f32, [8192]): 10%|▉ | 69/710 [00:00<00:00, 3446.31it/s] Saving model.layers.8.linear_attn.dt_bias (f32, [32]): 10%|▉ | 70/710 [00:00<00:00, 3488.24it/s] Saving model.layers.8.linear_attn.neg_exp_A (f32, [32]): 10%|█ | 71/710 [00:00<00:00, 3532.57it/s] Saving model.layers.8.linear_attn.norm.weight (f32, [128]): 10%|█ | 72/710 [00:00<00:00, 3541.49it/s] Saving model.layers.8.post_attention_layernorm.weight (f32, [2560]): 10%|█ | 73/710 [00:00<00:00, 3579.13it/s] Saving model.layers.9.input_layernorm.weight (f32, [2560]): 10%|█ | 74/710 [00:00<00:00, 3617.89it/s] Saving model.layers.9.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 11%|█ | 75/710 [00:00<00:00, 3656.85it/s] Saving model.layers.9.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 11%|█ | 76/710 [00:00<00:00, 3682.95it/s] Saving model.layers.9.linear_attn.conv1d.weight (f32, [8192,1,4]): 11%|█ | 77/710 [00:00<00:00, 3714.59it/s] Saving model.layers.9.linear_attn.conv1d.bias (f32, [8192]): 11%|█ | 78/710 [00:00<00:00, 3747.10it/s] Saving model.layers.9.linear_attn.dt_bias (f32, [32]): 11%|█ | 79/710 [00:00<00:00, 3775.21it/s] Saving model.layers.9.linear_attn.neg_exp_A (f32, [32]): 11%|█▏ | 80/710 [00:00<00:00, 3813.13it/s] Saving model.layers.9.linear_attn.norm.weight (f32, [128]): 11%|█▏ | 81/710 [00:00<00:00, 3853.00it/s] Saving model.layers.9.post_attention_layernorm.weight (f32, [2560]): 12%|█▏ | 82/710 [00:00<00:00, 3893.82it/s] Saving model.layers.10.input_layernorm.weight (f32, [2560]): 12%|█▏ | 83/710 [00:00<00:00, 3929.11it/s] Saving model.layers.10.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 12%|█▏ | 84/710 [00:00<00:00, 3967.14it/s] Saving model.layers.10.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 12%|█▏ | 85/710 [00:00<00:00, 4000.49it/s] Saving model.layers.10.linear_attn.conv1d.weight (f32, [8192,1,4]): 12%|█▏ | 86/710 [00:00<00:00, 4032.85it/s] Saving model.layers.10.linear_attn.conv1d.bias (f32, [8192]): 12%|█▏ | 87/710 [00:00<00:00, 4067.56it/s] Saving model.layers.10.linear_attn.dt_bias (f32, [32]): 12%|█▏ | 88/710 [00:00<00:00, 4107.17it/s] Saving model.layers.10.linear_attn.neg_exp_A (f32, [32]): 13%|█▎ | 89/710 [00:00<00:00, 4149.54it/s] Saving model.layers.10.linear_attn.norm.weight (f32, [128]): 13%|█▎ | 90/710 [00:00<00:00, 4191.46it/s] Saving model.layers.10.post_attention_layernorm.weight (f32, [2560]): 13%|█▎ | 91/710 [00:00<00:00, 4233.01it/s] Saving model.layers.11.input_layernorm.weight (f32, [2560]): 13%|█▎ | 92/710 [00:00<00:00, 4273.36it/s] Saving model.layers.11.attn.q_norm.layernorm.weight (f32, [256]): 13%|█▎ | 93/710 [00:00<00:00, 4313.74it/s] Saving model.layers.11.attn.k_norm.layernorm.weight (f32, [256]): 13%|█▎ | 94/710 [00:00<00:00, 4355.84it/s] Saving model.layers.11.post_attention_layernorm.weight (f32, [2560]): 13%|█▎ | 95/710 [00:00<00:00, 4397.22it/s] Saving model.layers.12.input_layernorm.weight (f32, [2560]): 14%|█▎ | 96/710 [00:00<00:00, 4437.10it/s] Saving model.layers.12.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 14%|█▎ | 97/710 [00:00<00:00, 4477.20it/s] Saving model.layers.12.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 14%|█▍ | 98/710 [00:00<00:00, 4509.31it/s] Saving model.layers.12.linear_attn.conv1d.weight (f32, [8192,1,4]): 14%|█▍ | 99/710 [00:00<00:00, 4541.18it/s] Saving model.layers.12.linear_attn.conv1d.bias (f32, [8192]): 14%|█▍ | 100/710 [00:00<00:00, 4575.39it/s] Saving model.layers.12.linear_attn.dt_bias (f32, [32]): 14%|█▍ | 101/710 [00:00<00:00, 4613.29it/s] Saving model.layers.12.linear_attn.neg_exp_A (f32, [32]): 14%|█▍ | 102/710 [00:00<00:00, 4654.30it/s] Saving model.layers.12.linear_attn.norm.weight (f32, [128]): 15%|█▍ | 103/710 [00:00<00:00, 4694.88it/s] Saving model.layers.12.post_attention_layernorm.weight (f32, [2560]): 15%|█▍ | 104/710 [00:00<00:00, 4735.47it/s] Saving model.layers.13.input_layernorm.weight (f32, [2560]): 15%|█▍ | 105/710 [00:00<00:00, 4774.47it/s] Saving model.layers.13.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 15%|█▍ | 106/710 [00:00<00:00, 4813.78it/s] Saving model.layers.13.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 15%|█▌ | 107/710 [00:00<00:00, 4843.15it/s] Saving model.layers.13.linear_attn.conv1d.weight (f32, [8192,1,4]): 15%|█▌ | 108/710 [00:00<00:00, 4872.27it/s] Saving model.layers.13.linear_attn.conv1d.bias (f32, [8192]): 15%|█▌ | 109/710 [00:00<00:00, 4904.83it/s] Saving model.layers.13.linear_attn.dt_bias (f32, [32]): 15%|█▌ | 110/710 [00:00<00:00, 4941.66it/s] Saving model.layers.13.linear_attn.neg_exp_A (f32, [32]): 16%|█▌ | 111/710 [00:00<00:00, 4981.84it/s] Saving model.layers.13.linear_attn.norm.weight (f32, [128]): 16%|█▌ | 112/710 [00:00<00:00, 5020.86it/s] Saving model.layers.13.post_attention_layernorm.weight (f32, [2560]): 16%|█▌ | 113/710 [00:00<00:00, 5060.34it/s] Saving model.layers.14.input_layernorm.weight (f32, [2560]): 16%|█▌ | 114/710 [00:00<00:00, 5097.72it/s] Saving model.layers.14.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 16%|█▌ | 115/710 [00:00<00:00, 5135.86it/s] Saving model.layers.14.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 16%|█▋ | 116/710 [00:00<00:00, 5163.75it/s] Saving model.layers.14.linear_attn.conv1d.weight (f32, [8192,1,4]): 16%|█▋ | 117/710 [00:00<00:00, 5190.37it/s] Saving model.layers.14.linear_attn.conv1d.bias (f32, [8192]): 17%|█▋ | 118/710 [00:00<00:00, 5221.69it/s] Saving model.layers.14.linear_attn.dt_bias (f32, [32]): 17%|█▋ | 119/710 [00:00<00:00, 5257.24it/s] Saving model.layers.14.linear_attn.neg_exp_A (f32, [32]): 17%|█▋ | 120/710 [00:00<00:00, 5296.23it/s] Saving model.layers.14.linear_attn.norm.weight (f32, [128]): 17%|█▋ | 121/710 [00:00<00:00, 5334.47it/s] Saving model.layers.14.post_attention_layernorm.weight (f32, [2560]): 17%|█▋ | 122/710 [00:00<00:00, 5373.08it/s] Saving model.layers.15.input_layernorm.weight (f32, [2560]): 17%|█▋ | 123/710 [00:00<00:00, 5410.02it/s] Saving model.layers.15.attn.q_norm.layernorm.weight (f32, [256]): 17%|█▋ | 124/710 [00:00<00:00, 5446.81it/s] Saving model.layers.15.attn.k_norm.layernorm.weight (f32, [256]): 18%|█▊ | 125/710 [00:00<00:00, 5485.56it/s] Saving model.layers.15.post_attention_layernorm.weight (f32, [2560]): 18%|█▊ | 126/710 [00:00<00:00, 5523.90it/s] Saving model.layers.16.input_layernorm.weight (f32, [2560]): 18%|█▊ | 127/710 [00:00<00:00, 5560.47it/s] Saving model.layers.16.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 18%|█▊ | 128/710 [00:00<00:00, 5597.48it/s] Saving model.layers.16.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 18%|█▊ | 129/710 [00:00<00:00, 5623.56it/s] Saving model.layers.16.linear_attn.conv1d.weight (f32, [8192,1,4]): 18%|█▊ | 130/710 [00:00<00:00, 5646.61it/s] Saving model.layers.16.linear_attn.conv1d.bias (f32, [8192]): 18%|█▊ | 131/710 [00:00<00:00, 5675.53it/s] Saving model.layers.16.linear_attn.dt_bias (f32, [32]): 19%|█▊ | 132/710 [00:00<00:00, 5709.71it/s] Saving model.layers.16.linear_attn.neg_exp_A (f32, [32]): 19%|█▊ | 133/710 [00:00<00:00, 5747.81it/s] Saving model.layers.16.linear_attn.norm.weight (f32, [128]): 19%|█▉ | 134/710 [00:00<00:00, 5785.25it/s] Saving model.layers.16.post_attention_layernorm.weight (f32, [2560]): 19%|█▉ | 135/710 [00:00<00:00, 5822.85it/s] Saving model.layers.17.input_layernorm.weight (f32, [2560]): 19%|█▉ | 136/710 [00:00<00:00, 5857.97it/s] Saving model.layers.17.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 19%|█▉ | 137/710 [00:00<00:00, 5893.60it/s] Saving model.layers.17.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 19%|█▉ | 138/710 [00:00<00:00, 5917.07it/s] Saving model.layers.17.linear_attn.conv1d.weight (f32, [8192,1,4]): 20%|█▉ | 139/710 [00:00<00:00, 5939.49it/s] Saving model.layers.17.linear_attn.conv1d.bias (f32, [8192]): 20%|█▉ | 140/710 [00:00<00:00, 5968.42it/s] Saving model.layers.17.linear_attn.dt_bias (f32, [32]): 20%|█▉ | 141/710 [00:00<00:00, 6001.84it/s] Saving model.layers.17.linear_attn.neg_exp_A (f32, [32]): 20%|██ | 142/710 [00:00<00:00, 6039.13it/s] Saving model.layers.17.linear_attn.norm.weight (f32, [128]): 20%|██ | 143/710 [00:00<00:00, 6075.62it/s] Saving model.layers.17.post_attention_layernorm.weight (f32, [2560]): 20%|██ | 144/710 [00:00<00:00, 6112.41it/s] Saving model.layers.18.input_layernorm.weight (f32, [2560]): 20%|██ | 145/710 [00:00<00:00, 6146.15it/s] Saving model.layers.18.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 21%|██ | 146/710 [00:00<00:00, 6180.67it/s] Saving model.layers.18.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 21%|██ | 147/710 [00:00<00:00, 6202.10it/s] Saving model.layers.18.linear_attn.conv1d.weight (f32, [8192,1,4]): 21%|██ | 148/710 [00:00<00:00, 6222.57it/s] Saving model.layers.18.linear_attn.conv1d.bias (f32, [8192]): 21%|██ | 149/710 [00:00<00:00, 6250.58it/s] Saving model.layers.18.linear_attn.dt_bias (f32, [32]): 21%|██ | 150/710 [00:00<00:00, 6282.72it/s] Saving model.layers.18.linear_attn.neg_exp_A (f32, [32]): 21%|██▏ | 151/710 [00:00<00:00, 6319.05it/s] Saving model.layers.18.linear_attn.norm.weight (f32, [128]): 21%|██▏ | 152/710 [00:00<00:00, 6354.69it/s] Saving model.layers.18.post_attention_layernorm.weight (f32, [2560]): 22%|██▏ | 153/710 [00:00<00:00, 6390.32it/s] Saving model.layers.19.input_layernorm.weight (f32, [2560]): 22%|██▏ | 154/710 [00:00<00:00, 6424.15it/s] Saving model.layers.19.attn.q_norm.layernorm.weight (f32, [256]): 22%|██▏ | 155/710 [00:00<00:00, 6458.10it/s] Saving model.layers.19.attn.k_norm.layernorm.weight (f32, [256]): 22%|██▏ | 156/710 [00:00<00:00, 6494.21it/s] Saving model.layers.19.post_attention_layernorm.weight (f32, [2560]): 22%|██▏ | 157/710 [00:00<00:00, 6529.43it/s] Saving model.layers.20.input_layernorm.weight (f32, [2560]): 22%|██▏ | 158/710 [00:00<00:00, 6561.78it/s] Saving model.layers.20.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 22%|██▏ | 159/710 [00:00<00:00, 6593.32it/s] Saving model.layers.20.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 23%|██▎ | 160/710 [00:00<00:00, 6611.84it/s] Saving model.layers.20.linear_attn.conv1d.weight (f32, [8192,1,4]): 23%|██▎ | 161/710 [00:00<00:00, 6631.60it/s] Saving model.layers.20.linear_attn.conv1d.bias (f32, [8192]): 23%|██▎ | 162/710 [00:00<00:00, 6658.08it/s] Saving model.layers.20.linear_attn.dt_bias (f32, [32]): 23%|██▎ | 163/710 [00:00<00:00, 6689.41it/s] Saving model.layers.20.linear_attn.neg_exp_A (f32, [32]): 23%|██▎ | 164/710 [00:00<00:00, 6724.86it/s] Saving model.layers.20.linear_attn.norm.weight (f32, [128]): 23%|██▎ | 165/710 [00:00<00:00, 6759.52it/s] Saving model.layers.20.post_attention_layernorm.weight (f32, [2560]): 23%|██▎ | 166/710 [00:00<00:00, 6794.18it/s] Saving model.layers.21.input_layernorm.weight (f32, [2560]): 24%|██▎ | 167/710 [00:00<00:00, 6826.99it/s] Saving model.layers.21.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 24%|██▎ | 168/710 [00:00<00:00, 6860.11it/s] Saving model.layers.21.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 24%|██▍ | 169/710 [00:00<00:00, 6882.52it/s] Saving model.layers.21.linear_attn.conv1d.weight (f32, [8192,1,4]): 24%|██▍ | 170/710 [00:00<00:00, 6904.47it/s] Saving model.layers.21.linear_attn.conv1d.bias (f32, [8192]): 24%|██▍ | 171/710 [00:00<00:00, 6930.32it/s] Saving model.layers.21.linear_attn.dt_bias (f32, [32]): 24%|██▍ | 172/710 [00:00<00:00, 6961.23it/s] Saving model.layers.21.linear_attn.neg_exp_A (f32, [32]): 24%|██▍ | 173/710 [00:00<00:00, 6995.76it/s] Saving model.layers.21.linear_attn.norm.weight (f32, [128]): 25%|██▍ | 174/710 [00:00<00:00, 7029.70it/s] Saving model.layers.21.post_attention_layernorm.weight (f32, [2560]): 25%|██▍ | 175/710 [00:00<00:00, 7063.16it/s] Saving model.layers.22.input_layernorm.weight (f32, [2560]): 25%|██▍ | 176/710 [00:00<00:00, 7095.05it/s] Saving model.layers.22.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 25%|██▍ | 177/710 [00:00<00:00, 7127.01it/s] Saving model.layers.22.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 25%|██▌ | 178/710 [00:00<00:00, 7148.74it/s] Saving model.layers.22.linear_attn.conv1d.weight (f32, [8192,1,4]): 25%|██▌ | 179/710 [00:00<00:00, 7168.11it/s] Saving model.layers.22.linear_attn.conv1d.bias (f32, [8192]): 25%|██▌ | 180/710 [00:00<00:00, 7192.50it/s] Saving model.layers.22.linear_attn.dt_bias (f32, [32]): 25%|██▌ | 181/710 [00:00<00:00, 7222.27it/s] Saving model.layers.22.linear_attn.neg_exp_A (f32, [32]): 26%|██▌ | 182/710 [00:00<00:00, 7253.75it/s] Saving model.layers.22.linear_attn.norm.weight (f32, [128]): 26%|██▌ | 183/710 [00:00<00:00, 7286.06it/s] Saving model.layers.22.post_attention_layernorm.weight (f32, [2560]): 26%|██▌ | 184/710 [00:00<00:00, 7319.28it/s] Saving model.layers.23.input_layernorm.weight (f32, [2560]): 26%|██▌ | 185/710 [00:00<00:00, 7350.27it/s] Saving model.layers.23.attn.q_norm.layernorm.weight (f32, [256]): 26%|██▌ | 186/710 [00:00<00:00, 7381.47it/s] Saving model.layers.23.attn.k_norm.layernorm.weight (f32, [256]): 26%|██▋ | 187/710 [00:00<00:00, 7414.98it/s] Saving model.layers.23.post_attention_layernorm.weight (f32, [2560]): 26%|██▋ | 188/710 [00:00<00:00, 7447.46it/s] Saving model.layers.24.input_layernorm.weight (f32, [2560]): 27%|██▋ | 189/710 [00:00<00:00, 7478.10it/s] Saving model.layers.24.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 27%|██▋ | 190/710 [00:00<00:00, 7509.24it/s] Saving model.layers.24.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 27%|██▋ | 191/710 [00:00<00:00, 7529.18it/s] Saving model.layers.24.linear_attn.conv1d.weight (f32, [8192,1,4]): 27%|██▋ | 192/710 [00:00<00:00, 7547.46it/s] Saving model.layers.24.linear_attn.conv1d.bias (f32, [8192]): 27%|██▋ | 193/710 [00:00<00:00, 7569.74it/s] Saving model.layers.24.linear_attn.dt_bias (f32, [32]): 27%|██▋ | 194/710 [00:00<00:00, 7598.02it/s] Saving model.layers.24.linear_attn.neg_exp_A (f32, [32]): 27%|██▋ | 195/710 [00:00<00:00, 7630.77it/s] Saving model.layers.24.linear_attn.norm.weight (f32, [128]): 28%|██▊ | 196/710 [00:00<00:00, 7662.83it/s] Saving model.layers.24.post_attention_layernorm.weight (f32, [2560]): 28%|██▊ | 197/710 [00:00<00:00, 7695.40it/s] Saving model.layers.25.input_layernorm.weight (f32, [2560]): 28%|██▊ | 198/710 [00:00<00:00, 7724.75it/s] Saving model.layers.25.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 28%|██▊ | 199/710 [00:00<00:00, 7755.11it/s] Saving model.layers.25.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 28%|██▊ | 200/710 [00:00<00:00, 7775.29it/s] Saving model.layers.25.linear_attn.conv1d.weight (f32, [8192,1,4]): 28%|██▊ | 201/710 [00:00<00:00, 7794.37it/s] Saving model.layers.25.linear_attn.conv1d.bias (f32, [8192]): 28%|██▊ | 202/710 [00:00<00:00, 7816.53it/s] Saving model.layers.25.linear_attn.dt_bias (f32, [32]): 29%|██▊ | 203/710 [00:00<00:00, 7843.94it/s] Saving model.layers.25.linear_attn.neg_exp_A (f32, [32]): 29%|██▊ | 204/710 [00:00<00:00, 7876.48it/s] Saving model.layers.25.linear_attn.norm.weight (f32, [128]): 29%|██▉ | 205/710 [00:00<00:00, 7908.18it/s] Saving model.layers.25.post_attention_layernorm.weight (f32, [2560]): 29%|██▉ | 206/710 [00:00<00:00, 7939.89it/s] Saving model.layers.26.input_layernorm.weight (f32, [2560]): 29%|██▉ | 207/710 [00:00<00:00, 7969.20it/s] Saving model.layers.26.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 29%|██▉ | 208/710 [00:00<00:00, 7998.75it/s] Saving model.layers.26.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 29%|██▉ | 209/710 [00:00<00:00, 8016.62it/s] Saving model.layers.26.linear_attn.conv1d.weight (f32, [8192,1,4]): 30%|██▉ | 210/710 [00:00<00:00, 8033.16it/s] Saving model.layers.26.linear_attn.conv1d.bias (f32, [8192]): 30%|██▉ | 211/710 [00:00<00:00, 8054.22it/s] Saving model.layers.26.linear_attn.dt_bias (f32, [32]): 30%|██▉ | 212/710 [00:00<00:00, 8081.07it/s] Saving model.layers.26.linear_attn.neg_exp_A (f32, [32]): 30%|███ | 213/710 [00:00<00:00, 8112.85it/s] Saving model.layers.26.linear_attn.norm.weight (f32, [128]): 30%|███ | 214/710 [00:00<00:00, 8143.91it/s] Saving model.layers.26.post_attention_layernorm.weight (f32, [2560]): 30%|███ | 215/710 [00:00<00:00, 8175.29it/s] Saving model.layers.27.input_layernorm.weight (f32, [2560]): 30%|███ | 216/710 [00:00<00:00, 8204.31it/s] Saving model.layers.27.attn.q_norm.layernorm.weight (f32, [256]): 31%|███ | 217/710 [00:00<00:00, 8233.35it/s] Saving model.layers.27.attn.k_norm.layernorm.weight (f32, [256]): 31%|███ | 218/710 [00:00<00:00, 8264.94it/s] Saving model.layers.27.post_attention_layernorm.weight (f32, [2560]): 31%|███ | 219/710 [00:00<00:00, 8295.73it/s] Saving model.layers.28.input_layernorm.weight (f32, [2560]): 31%|███ | 220/710 [00:00<00:00, 8324.28it/s] Saving model.layers.28.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 31%|███ | 221/710 [00:00<00:00, 8353.30it/s] Saving model.layers.28.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 31%|███▏ | 222/710 [00:00<00:00, 8372.62it/s] Saving model.layers.28.linear_attn.conv1d.weight (f32, [8192,1,4]): 31%|███▏ | 223/710 [00:00<00:00, 8389.74it/s] Saving model.layers.28.linear_attn.conv1d.bias (f32, [8192]): 32%|███▏ | 224/710 [00:00<00:00, 8411.14it/s] Saving model.layers.28.linear_attn.dt_bias (f32, [32]): 32%|███▏ | 225/710 [00:00<00:00, 8437.51it/s] Saving model.layers.28.linear_attn.neg_exp_A (f32, [32]): 32%|███▏ | 226/710 [00:00<00:00, 8467.89it/s] Saving model.layers.28.linear_attn.norm.weight (f32, [128]): 32%|███▏ | 227/710 [00:00<00:00, 8498.07it/s] Saving model.layers.28.post_attention_layernorm.weight (f32, [2560]): 32%|███▏ | 228/710 [00:00<00:00, 8528.43it/s] Saving model.layers.29.input_layernorm.weight (f32, [2560]): 32%|███▏ | 229/710 [00:00<00:00, 8556.22it/s] Saving model.layers.29.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 32%|███▏ | 230/710 [00:00<00:00, 8584.18it/s] Saving model.layers.29.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 33%|███▎ | 231/710 [00:00<00:00, 8601.37it/s] Saving model.layers.29.linear_attn.conv1d.weight (f32, [8192,1,4]): 33%|███▎ | 232/710 [00:00<00:00, 8614.75it/s] Saving model.layers.29.linear_attn.conv1d.bias (f32, [8192]): 33%|███▎ | 233/710 [00:00<00:00, 8635.06it/s] Saving model.layers.29.linear_attn.dt_bias (f32, [32]): 33%|███▎ | 234/710 [00:00<00:00, 8660.72it/s] Saving model.layers.29.linear_attn.neg_exp_A (f32, [32]): 33%|███▎ | 235/710 [00:00<00:00, 8691.36it/s] Saving model.layers.29.linear_attn.norm.weight (f32, [128]): 33%|███▎ | 236/710 [00:00<00:00, 8721.12it/s] Saving model.layers.29.post_attention_layernorm.weight (f32, [2560]): 33%|███▎ | 237/710 [00:00<00:00, 8750.44it/s] Saving model.layers.30.input_layernorm.weight (f32, [2560]): 34%|███▎ | 238/710 [00:00<00:00, 8776.78it/s] Saving model.layers.30.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 34%|███▎ | 239/710 [00:00<00:00, 8804.29it/s] Saving model.layers.30.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 34%|███▍ | 240/710 [00:00<00:00, 8817.04it/s] Saving model.layers.30.linear_attn.conv1d.weight (f32, [8192,1,4]): 34%|███▍ | 241/710 [00:00<00:00, 8827.80it/s] Saving model.layers.30.linear_attn.conv1d.bias (f32, [8192]): 34%|███▍ | 242/710 [00:00<00:00, 8847.12it/s] Saving model.layers.30.linear_attn.dt_bias (f32, [32]): 34%|███▍ | 243/710 [00:00<00:00, 8871.77it/s] Saving model.layers.30.linear_attn.neg_exp_A (f32, [32]): 34%|███▍ | 244/710 [00:00<00:00, 8901.70it/s] Saving model.layers.30.linear_attn.norm.weight (f32, [128]): 35%|███▍ | 245/710 [00:00<00:00, 8931.03it/s] Saving model.layers.30.post_attention_layernorm.weight (f32, [2560]): 35%|███▍ | 246/710 [00:00<00:00, 8960.32it/s] Saving model.layers.31.input_layernorm.weight (f32, [2560]): 35%|███▍ | 247/710 [00:00<00:00, 8987.07it/s] Saving model.layers.31.attn.q_norm.layernorm.weight (f32, [256]): 35%|███▍ | 248/710 [00:00<00:00, 9014.61it/s] Saving model.layers.31.attn.k_norm.layernorm.weight (f32, [256]): 35%|███▌ | 249/710 [00:00<00:00, 9043.91it/s] Saving model.layers.31.post_attention_layernorm.weight (f32, [2560]): 35%|███▌ | 250/710 [00:00<00:00, 9072.45it/s] Saving model.layers.32.final_norm_layernorm.weight (f32, [2560]): 35%|███▌ | 251/710 [00:00<00:00, 9099.37it/s] Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 35%|███▌ | 252/710 [00:00<00:00, 9126.24it/s] Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 36%|███▌ | 253/710 [00:00<00:00, 8954.32it/s] Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 36%|███▌ | 254/710 [00:00<00:00, 8951.65it/s] Saving model.layers.0.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 36%|███▌ | 255/710 [00:00<00:00, 8572.98it/s] Saving model.layers.0.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 36%|███▌ | 256/710 [00:00<00:00, 8506.50it/s] Saving model.layers.0.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 36%|███▌ | 257/710 [00:00<00:00, 8515.98it/s] Saving model.layers.0.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 36%|███▋ | 258/710 [00:00<00:00, 8375.62it/s] Saving model.layers.0.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 36%|███▋ | 259/710 [00:00<00:00, 8328.48it/s] Saving model.layers.0.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 37%|███▋ | 260/710 [00:00<00:00, 8340.17it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 37%|███▋ | 261/710 [00:00<00:00, 8219.00it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 37%|███▋ | 262/710 [00:00<00:00, 7914.75it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 37%|███▋ | 263/710 [00:00<00:00, 7895.77it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 37%|███▋ | 264/710 [00:00<00:00, 7257.06it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 37%|███▋ | 265/710 [00:00<00:00, 7132.62it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 37%|███▋ | 266/710 [00:00<00:00, 7133.49it/s] Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 38%|███▊ | 267/710 [00:00<00:00, 6890.80it/s] Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 38%|███▊ | 268/710 [00:00<00:00, 6798.60it/s] Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 38%|███▊ | 269/710 [00:00<00:00, 6804.71it/s] Saving model.layers.1.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 38%|███▊ | 270/710 [00:00<00:00, 6641.85it/s] Saving model.layers.1.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 38%|███▊ | 271/710 [00:00<00:00, 6603.40it/s] Saving model.layers.1.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 38%|███▊ | 272/710 [00:00<00:00, 6611.25it/s] Saving model.layers.1.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 38%|███▊ | 273/710 [00:00<00:00, 6538.93it/s] Saving model.layers.1.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 39%|███▊ | 274/710 [00:00<00:00, 6520.84it/s] Saving model.layers.1.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 39%|███▊ | 275/710 [00:00<00:00, 6533.22it/s] Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 39%|███▉ | 276/710 [00:00<00:00, 6471.32it/s] Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 39%|███▉ | 277/710 [00:00<00:00, 6272.15it/s] Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 39%|███▉ | 278/710 [00:00<00:00, 6264.63it/s] Saving model.layers.1.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 39%|███▉ | 279/710 [00:00<00:00, 5857.53it/s] Saving model.layers.1.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 39%|███▉ | 280/710 [00:00<00:00, 5778.67it/s] Saving model.layers.1.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 40%|███▉ | 281/710 [00:00<00:00, 5785.76it/s] Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 40%|███▉ | 282/710 [00:00<00:00, 5551.82it/s] Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 40%|███▉ | 283/710 [00:00<00:00, 3062.30it/s] Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 40%|████ | 284/710 [00:00<00:00, 3067.99it/s] Saving model.layers.2.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 40%|████ | 285/710 [00:00<00:00, 3037.98it/s] Saving model.layers.2.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 40%|████ | 286/710 [00:00<00:00, 3037.00it/s] Saving model.layers.2.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 40%|████ | 287/710 [00:00<00:00, 3045.04it/s] Saving model.layers.2.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 41%|████ | 288/710 [00:00<00:00, 3040.12it/s] Saving model.layers.2.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 41%|████ | 289/710 [00:00<00:00, 3038.54it/s] Saving model.layers.2.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 41%|████ | 290/710 [00:00<00:00, 3046.43it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 41%|████ | 291/710 [00:00<00:00, 3037.61it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 41%|████ | 292/710 [00:00<00:00, 2999.86it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 41%|████▏ | 293/710 [00:00<00:00, 3003.00it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 41%|████▏ | 294/710 [00:00<00:00, 2921.53it/s] Saving model.layers.2.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 41%|████▏ | 294/710 [00:00<00:00, 2921.53it/s] Saving model.layers.2.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 42%|████▏ | 295/710 [00:00<00:00, 2921.53it/s] Saving model.layers.2.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 42%|████▏ | 296/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 42%|████▏ | 297/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 42%|████▏ | 298/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 42%|████▏ | 299/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 42%|████▏ | 300/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 42%|████▏ | 301/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 43%|████▎ | 302/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 43%|████▎ | 303/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 43%|████▎ | 304/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 43%|████▎ | 305/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 43%|████▎ | 306/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 43%|████▎ | 307/710 [00:00<00:00, 2921.53it/s] Saving model.layers.3.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 43%|████▎ | 308/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 44%|████▎ | 309/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 44%|████▎ | 310/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 44%|████▍ | 311/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 44%|████▍ | 312/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 44%|████▍ | 313/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 44%|████▍ | 314/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 44%|████▍ | 315/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 45%|████▍ | 316/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 45%|████▍ | 317/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 45%|████▍ | 318/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 45%|████▍ | 319/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 45%|████▌ | 320/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 45%|████▌ | 321/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 45%|████▌ | 322/710 [00:00<00:00, 2921.53it/s] Saving model.layers.4.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 45%|████▌ | 323/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 46%|████▌ | 324/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 46%|████▌ | 325/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 46%|████▌ | 326/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 46%|████▌ | 327/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 46%|████▌ | 328/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 46%|████▋ | 329/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 46%|████▋ | 330/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 47%|████▋ | 331/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 47%|████▋ | 332/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 47%|████▋ | 333/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 47%|████▋ | 334/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 47%|████▋ | 335/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 47%|████▋ | 336/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 47%|████▋ | 337/710 [00:00<00:00, 2921.53it/s] Saving model.layers.5.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 48%|████▊ | 338/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 48%|████▊ | 339/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 48%|████▊ | 340/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 48%|████▊ | 341/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 48%|████▊ | 342/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 48%|████▊ | 343/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 48%|████▊ | 344/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 49%|████▊ | 345/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 49%|████▊ | 346/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 49%|████▉ | 347/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 49%|████▉ | 348/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 49%|████▉ | 349/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 49%|████▉ | 350/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 49%|████▉ | 351/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 50%|████▉ | 352/710 [00:00<00:00, 2921.53it/s] Saving model.layers.6.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 50%|████▉ | 353/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 50%|████▉ | 354/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 50%|█████ | 355/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 50%|█████ | 356/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 50%|█████ | 357/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 50%|█████ | 358/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 51%|█████ | 359/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 51%|█████ | 360/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 51%|█████ | 361/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 51%|█████ | 362/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 51%|█████ | 363/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 51%|█████▏ | 364/710 [00:00<00:00, 2921.53it/s] Saving model.layers.7.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 51%|█████▏ | 365/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 52%|█████▏ | 366/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 52%|█████▏ | 367/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 52%|█████▏ | 368/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 52%|█████▏ | 369/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 52%|█████▏ | 370/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 52%|█████▏ | 371/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 52%|█████▏ | 372/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 53%|█████▎ | 373/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 53%|█████▎ | 374/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 53%|█████▎ | 375/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 53%|█████▎ | 376/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 53%|█████▎ | 377/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 53%|█████▎ | 378/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 53%|█████▎ | 379/710 [00:00<00:00, 2921.53it/s] Saving model.layers.8.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 54%|█████▎ | 380/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 54%|█████▎ | 381/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 54%|█████▍ | 382/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 54%|█████▍ | 383/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 54%|█████▍ | 384/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 54%|█████▍ | 385/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 54%|█████▍ | 386/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 55%|█████▍ | 387/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 55%|█████▍ | 388/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 55%|█████▍ | 389/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 55%|█████▍ | 390/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 55%|█████▌ | 391/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 55%|█████▌ | 392/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 55%|█████▌ | 393/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 55%|█████▌ | 394/710 [00:00<00:00, 2921.53it/s] Saving model.layers.9.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 56%|█████▌ | 395/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 56%|█████▌ | 396/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 56%|█████▌ | 397/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 56%|█████▌ | 398/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 56%|█████▌ | 399/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 56%|█████▋ | 400/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 56%|█████▋ | 401/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 57%|█████▋ | 402/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 57%|█████▋ | 403/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 57%|█████▋ | 404/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 57%|█████▋ | 405/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 57%|█████▋ | 406/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 57%|█████▋ | 407/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 57%|█████▋ | 408/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 58%|█████▊ | 409/710 [00:00<00:00, 2921.53it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 58%|█████▊ | 410/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 58%|█████▊ | 411/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 58%|█████▊ | 412/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 58%|█████▊ | 413/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 58%|█████▊ | 414/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 58%|█████▊ | 415/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 59%|█████▊ | 416/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 59%|█████▊ | 417/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 59%|█████▉ | 418/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 59%|█████▉ | 419/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 59%|█████▉ | 420/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 59%|█████▉ | 421/710 [00:00<00:00, 2921.53it/s] Saving model.layers.11.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 59%|█████▉ | 422/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 60%|█████▉ | 423/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 60%|█████▉ | 424/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 60%|█████▉ | 425/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 60%|██████ | 426/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 60%|██████ | 427/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 60%|██████ | 428/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 60%|██████ | 429/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 61%|██████ | 430/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 61%|██████ | 431/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 61%|██████ | 432/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 61%|██████ | 433/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 61%|██████ | 434/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 61%|██████▏ | 435/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 61%|██████▏ | 436/710 [00:00<00:00, 2921.53it/s] Saving model.layers.12.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 62%|██████▏ | 437/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 62%|██████▏ | 438/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 62%|██████▏ | 439/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 62%|██████▏ | 440/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 62%|██████▏ | 441/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 62%|██████▏ | 442/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 62%|██████▏ | 443/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 63%|██████▎ | 444/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 63%|██████▎ | 445/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 63%|██████▎ | 446/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 63%|██████▎ | 447/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 63%|██████▎ | 448/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 63%|██████▎ | 449/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 63%|██████▎ | 450/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 64%|██████▎ | 451/710 [00:00<00:00, 2921.53it/s] Saving model.layers.13.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 64%|██████▎ | 452/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 64%|██████▍ | 453/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 64%|██████▍ | 454/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 64%|██████▍ | 455/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 64%|██████▍ | 456/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 64%|██████▍ | 457/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 65%|██████▍ | 458/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 65%|██████▍ | 459/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 65%|██████▍ | 460/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 65%|██████▍ | 461/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 65%|██████▌ | 462/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 65%|██████▌ | 463/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 65%|██████▌ | 464/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 65%|██████▌ | 465/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 66%|██████▌ | 466/710 [00:00<00:00, 2921.53it/s] Saving model.layers.14.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 66%|██████▌ | 467/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 66%|██████▌ | 468/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 66%|██████▌ | 469/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 66%|██████▌ | 470/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 66%|██████▋ | 471/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 66%|██████▋ | 472/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 67%|██████▋ | 473/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 67%|██████▋ | 474/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 67%|██████▋ | 475/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 67%|██████▋ | 476/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 67%|██████▋ | 477/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 67%|██████▋ | 478/710 [00:00<00:00, 2921.53it/s] Saving model.layers.15.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 67%|██████▋ | 479/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 68%|██████▊ | 480/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 68%|██████▊ | 481/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 68%|██████▊ | 482/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 68%|██████▊ | 483/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 68%|██████▊ | 484/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 68%|██████▊ | 485/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 68%|██████▊ | 486/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 69%|██████▊ | 487/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 69%|██████▊ | 488/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 69%|██████▉ | 489/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 69%|██████▉ | 490/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 69%|██████▉ | 491/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 69%|██████▉ | 492/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 69%|██████▉ | 493/710 [00:00<00:00, 2921.53it/s] Saving model.layers.16.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 70%|██████▉ | 494/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 70%|██████▉ | 495/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 70%|██████▉ | 496/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 70%|███████ | 497/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 70%|███████ | 498/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 70%|███████ | 499/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 70%|███████ | 500/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 71%|███████ | 501/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 71%|███████ | 502/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 71%|███████ | 503/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 71%|███████ | 504/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 71%|███████ | 505/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 71%|███████▏ | 506/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 71%|███████▏ | 507/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 72%|███████▏ | 508/710 [00:00<00:00, 2921.53it/s] Saving model.layers.17.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 72%|███████▏ | 509/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 72%|███████▏ | 510/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 72%|███████▏ | 511/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 72%|███████▏ | 512/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 72%|███████▏ | 513/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 72%|███████▏ | 514/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 73%|███████▎ | 515/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 73%|███████▎ | 516/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 73%|███████▎ | 517/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 73%|███████▎ | 518/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 73%|███████▎ | 519/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 73%|███████▎ | 520/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 73%|███████▎ | 521/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 74%|███████▎ | 522/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 74%|███████▎ | 523/710 [00:00<00:00, 2921.53it/s] Saving model.layers.18.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 74%|███████▍ | 524/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 74%|███████▍ | 525/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 74%|███████▍ | 526/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 74%|███████▍ | 527/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 74%|███████▍ | 528/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 75%|███████▍ | 529/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 75%|███████▍ | 530/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 75%|███████▍ | 531/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 75%|███████▍ | 532/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 75%|███████▌ | 533/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 75%|███████▌ | 534/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 75%|███████▌ | 535/710 [00:00<00:00, 2921.53it/s] Saving model.layers.19.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 75%|███████▌ | 536/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 76%|███████▌ | 537/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 76%|███████▌ | 538/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 76%|███████▌ | 539/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 76%|███████▌ | 540/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 76%|███████▌ | 541/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 76%|███████▋ | 542/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 76%|███████▋ | 543/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 77%|███████▋ | 544/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 77%|███████▋ | 545/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 77%|███████▋ | 546/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 77%|███████▋ | 547/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 77%|███████▋ | 548/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 77%|███████▋ | 549/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 77%|███████▋ | 550/710 [00:00<00:00, 2921.53it/s] Saving model.layers.20.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 78%|███████▊ | 551/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 78%|███████▊ | 552/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 78%|███████▊ | 553/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 78%|███████▊ | 554/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 78%|███████▊ | 555/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 78%|███████▊ | 556/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 78%|███████▊ | 557/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 79%|███████▊ | 558/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 79%|███████▊ | 559/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 79%|███████▉ | 560/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 79%|███████▉ | 561/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 79%|███████▉ | 562/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 79%|███████▉ | 563/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 79%|███████▉ | 564/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 80%|███████▉ | 565/710 [00:00<00:00, 2921.53it/s] Saving model.layers.21.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 80%|███████▉ | 566/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 80%|███████▉ | 567/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 80%|████████ | 568/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 80%|████████ | 569/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 80%|████████ | 570/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 80%|████████ | 571/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 81%|████████ | 572/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 81%|████████ | 573/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 81%|████████ | 574/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 81%|████████ | 575/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 81%|████████ | 576/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 81%|████████▏ | 577/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 81%|████████▏ | 578/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 82%|████████▏ | 579/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 82%|████████▏ | 580/710 [00:00<00:00, 2921.53it/s] Saving model.layers.22.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 82%|████████▏ | 581/710 [00:00<00:00, 2921.53it/s] Saving model.layers.23.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 82%|████████▏ | 582/710 [00:00<00:00, 2921.53it/s] Saving model.layers.23.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 82%|████████▏ | 583/710 [00:00<00:00, 2921.53it/s] Saving model.layers.23.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 82%|████████▏ | 584/710 [00:00<00:00, 2921.53it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 82%|████████▏ | 585/710 [00:00<00:00, 2921.53it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 83%|████████▎ | 586/710 [00:00<00:00, 2921.53it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 83%|████████▎ | 587/710 [00:00<00:00, 800.80it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 83%|████████▎ | 587/710 [00:00<00:00, 800.80it/s] Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 83%|████████▎ | 588/710 [00:00<00:00, 800.80it/s] Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 83%|████████▎ | 589/710 [00:00<00:00, 800.80it/s] Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 83%|████████▎ | 590/710 [00:00<00:00, 800.80it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 83%|████████▎ | 591/710 [00:00<00:00, 800.80it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 83%|████████▎ | 592/710 [00:00<00:00, 800.80it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 84%|████████▎ | 593/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 84%|████████▎ | 594/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 84%|████████▍ | 595/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 84%|████████▍ | 596/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 84%|████████▍ | 597/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 84%|████████▍ | 598/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 84%|████████▍ | 599/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 85%|████████▍ | 600/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 85%|████████▍ | 601/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 85%|████████▍ | 602/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 85%|████████▍ | 603/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 85%|████████▌ | 604/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 85%|████████▌ | 605/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 85%|████████▌ | 606/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 85%|████████▌ | 607/710 [00:00<00:00, 800.80it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 86%|████████▌ | 608/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 86%|████████▌ | 609/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 86%|████████▌ | 610/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 86%|████████▌ | 611/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 86%|████████▌ | 612/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 86%|████████▋ | 613/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 86%|████████▋ | 614/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 87%|████████▋ | 615/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 87%|████████▋ | 616/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 87%|████████▋ | 617/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 87%|████████▋ | 618/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 87%|████████▋ | 619/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 87%|████████▋ | 620/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 87%|████████▋ | 621/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 88%|████████▊ | 622/710 [00:00<00:00, 800.80it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 88%|████████▊ | 623/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 88%|████████▊ | 624/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 88%|████████▊ | 625/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 88%|████████▊ | 626/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 88%|████████▊ | 627/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 88%|████████▊ | 628/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 89%|████████▊ | 629/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 89%|████████▊ | 630/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 89%|████████▉ | 631/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 89%|████████▉ | 632/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 89%|████████▉ | 633/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 89%|████████▉ | 634/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 89%|████████▉ | 635/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 90%|████████▉ | 636/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 90%|████████▉ | 637/710 [00:00<00:00, 800.80it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 90%|████████▉ | 638/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 90%|█████████ | 639/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 90%|█████████ | 640/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 90%|█████████ | 641/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 90%|█████████ | 642/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 91%|█████████ | 643/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 91%|█████████ | 644/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 91%|█████████ | 645/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 91%|█████████ | 646/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 91%|█████████ | 647/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 91%|█████████▏| 648/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 91%|█████████▏| 649/710 [00:00<00:00, 800.80it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 92%|█████████▏| 650/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 92%|█████████▏| 651/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 92%|█████████▏| 652/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 92%|█████████▏| 653/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 92%|█████████▏| 654/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 92%|█████████▏| 655/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 92%|█████████▏| 656/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 93%|█████████▎| 657/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 93%|█████████▎| 658/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 93%|█████████▎| 659/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 93%|█████████▎| 660/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 93%|█████████▎| 661/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 93%|█████████▎| 662/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 93%|█████████▎| 663/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 94%|█████████▎| 664/710 [00:00<00:00, 800.80it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 94%|█████████▎| 665/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 94%|█████████▍| 666/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 94%|█████████▍| 667/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 94%|█████████▍| 668/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 94%|█████████▍| 669/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 94%|█████████▍| 670/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 95%|█████████▍| 671/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 95%|█████████▍| 672/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 95%|█████████▍| 673/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 95%|█████████▍| 674/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 95%|█████████▌| 675/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 95%|█████████▌| 676/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 95%|█████████▌| 677/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 95%|█████████▌| 678/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 96%|█████████▌| 679/710 [00:00<00:00, 800.80it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 96%|█████████▌| 680/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_scale (f32, [8192,160]): 96%|█████████▌| 681/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,80]): 96%|█████████▌| 682/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_Q4G16 (u8, [8192,160,8]): 96%|█████████▌| 683/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.z_proj.MatMul.weight_scale (f32, [4096,160]): 96%|█████████▋| 684/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,80]): 96%|█████████▋| 685/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.z_proj.MatMul.weight_Q4G16 (u8, [4096,160,8]): 97%|█████████▋| 686/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.out_proj.MatMul.weight_scale (f32, [2560,256]): 97%|█████████▋| 687/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,128]): 97%|█████████▋| 688/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.linear_attn.out_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 97%|█████████▋| 689/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 97%|█████████▋| 690/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 97%|█████████▋| 691/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 97%|█████████▋| 692/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 98%|█████████▊| 693/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 98%|█████████▊| 694/710 [00:00<00:00, 800.80it/s] Saving model.layers.30.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 98%|█████████▊| 695/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 98%|█████████▊| 696/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 98%|█████████▊| 697/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 98%|█████████▊| 698/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 98%|█████████▊| 699/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 99%|█████████▊| 700/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 99%|█████████▊| 701/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 99%|█████████▉| 702/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 99%|█████████▉| 703/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 99%|█████████▉| 704/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 99%|█████████▉| 705/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 99%|█████████▉| 706/710 [00:00<00:00, 800.80it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 100%|█████████▉| 707/710 [00:00<00:00, 800.80it/s] Saving lm_head.MatMul.weight_scale (f32, [248320,160]): 100%|█████████▉| 708/710 [00:00<00:00, 800.80it/s] Saving lm_head.MatMul.weight_zp (u8, [248320,80]): 100%|█████████▉| 709/710 [00:00<00:00, 800.80it/s] Saving lm_head.MatMul.weight_Q4G16 (u8, [248320,160,8]): 100%|██████████| 710/710 [00:00<00:00, 800.80it/s] Saving lm_head.MatMul.weight_Q4G16 (u8, [248320,160,8]): 100%|██████████| 710/710 [00:01<00:00, 645.84it/s]
2026-11-06 01:10:16,153 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize model with k_quant algorithm...
2026-11-06 01:10:16,154 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_Q4G16' of node '/model/mtp/embed_tokens/Reshape' (::Reshape:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-06 01:10:16,154 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_scale' of node '/model/mtp/embed_tokens/GatherBlockQuantized' (com.microsoft::GatherBlockQuantized:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-06 01:10:16,154 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_zp' of node '/model/mtp/embed_tokens/GatherBlockQuantized' (com.microsoft::GatherBlockQuantized:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-06 01:10:16,154 neural_compressor [WARNING] - Model size > 2GB. Please use model path instead of onnx model object to quantize
2026-11-06 01:10:16,167 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:10:18,547 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:10:23,203 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:10:25,143 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:10:33,430 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:10:38,161 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-06 01:14:08,441 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of model with k_quant algorithm.
Saving ONNX model in model-cpu-new
Progress: [### ] 16.67% Progress: [###### ] 33.33% Progress: [########## ] 50.00% Progress: [############# ] 66.67% Progress: [################ ] 83.33% Progress: [####################] 100.00% 0it [00:00, ?it/s] Saving model.mtp.pre_fc_norm_embedding.weight (f32, [2560]): 4%|▎ | 1/27 [00:00<00:00, 622.95it/s] Saving model.mtp.pre_fc_norm_hidden.weight (f32, [2560]): 7%|▋ | 2/27 [00:00<00:00, 1183.16it/s] Saving model.layers.0.input_layernorm.weight (f32, [2560]): 11%|█ | 3/27 [00:00<00:00, 1731.99it/s] Saving model.layers.0.attn.q_norm.layernorm.weight (f32, [256]): 15%|█▍ | 4/27 [00:00<00:00, 2265.97it/s] Saving model.layers.0.attn.k_norm.layernorm.weight (f32, [256]): 19%|█▊ | 5/27 [00:00<00:00, 2793.97it/s] Saving cos_cache (f32, [262144,32]): 22%|██▏ | 6/27 [00:00<00:00, 3306.07it/s] Saving sin_cache (f32, [262144,32]): 26%|██▌ | 7/27 [00:00<00:00, 881.42it/s] Saving model.layers.0.post_attention_layernorm.weight (f32, [2560]): 30%|██▉ | 8/27 [00:00<00:00, 659.70it/s] Saving model.layers.1.final_norm_layernorm.weight (f32, [2560]): 33%|███▎ | 9/27 [00:00<00:00, 723.96it/s] Saving model.mtp.fc.MatMul.weight_scale (f32, [2560,320]): 37%|███▋ | 10/27 [00:00<00:00, 801.43it/s] Saving model.mtp.fc.MatMul.weight_zp (u8, [2560,160]): 41%|████ | 11/27 [00:00<00:00, 848.72it/s] Saving model.mtp.fc.MatMul.weight_Q4G16 (u8, [2560,320,8]): 44%|████▍ | 12/27 [00:00<00:00, 917.76it/s] Saving model.layers.0.attn.qkv_proj.MatMul.weight_scale (f32, [10240,160]): 48%|████▊ | 13/27 [00:00<00:00, 918.18it/s] Saving model.layers.0.attn.qkv_proj.MatMul.weight_zp (u8, [10240,80]): 52%|█████▏ | 14/27 [00:00<00:00, 934.47it/s] Saving model.layers.0.attn.qkv_proj.MatMul.weight_Q4G16 (u8, [10240,160,8]): 56%|█████▌ | 15/27 [00:00<00:00, 990.22it/s] Saving model.layers.0.attn.o_proj.MatMul.weight_scale (f32, [2560,256]): 59%|█████▉ | 16/27 [00:00<00:00, 942.06it/s] Saving model.layers.0.attn.o_proj.MatMul.weight_zp (u8, [2560,128]): 63%|██████▎ | 17/27 [00:00<00:00, 979.22it/s] Saving model.layers.0.attn.o_proj.MatMul.weight_Q4G16 (u8, [2560,256,8]): 67%|██████▋ | 18/27 [00:00<00:00, 1031.63it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scale (f32, [18432,160]): 70%|███████ | 19/27 [00:00<00:00, 1049.02it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,80]): 74%|███████▍ | 20/27 [00:00<00:00, 1023.10it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_Q4G16 (u8, [18432,160,8]): 78%|███████▊ | 21/27 [00:00<00:00, 1063.14it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_scale (f32, [2560,576]): 81%|████████▏ | 22/27 [00:00<00:00, 966.34it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_zp (u8, [2560,288]): 85%|████████▌ | 23/27 [00:00<00:00, 976.28it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_Q4G16 (u8, [2560,576,8]): 89%|████████▉ | 24/27 [00:00<00:00, 1013.41it/s] Saving lm_head.MatMul.weight_scale (f32, [248320,160]): 93%|█████████▎| 25/27 [00:00<00:00, 994.72it/s] Saving lm_head.MatMul.weight_zp (u8, [248320,80]): 96%|█████████▋| 26/27 [00:00<00:00, 348.07it/s] Saving lm_head.MatMul.weight_Q4G16 (u8, [248320,160,8]): 100%|██████████| 27/27 [00:00<00:00, 349.12it/s] Saving lm_head.MatMul.weight_Q4G16 (u8, [248320,160,8]): 100%|██████████| 27/27 [00:00<00:00, 128.22it/s]
Configured 497 MB of runtime initializer sharing; model-cpu-new/mtp.onnx.data remains self-contained.
Warning: could not read generation_config.json (/home/anon/LLM/KnowLine-4B-Gen5.5 does not appear to have a file named generation_config.json. Checkout 'https://huggingface.co//home/anon/LLM/KnowLine-4B-Gen5.5/tree/main' for available files.). Falling back to config.json.
Adding the tokenizer's end-of-turn token <|im_end|> (id 248046) to eos_token_id from the model configuration: 248044.
Saving GenAI config in model-cpu-new
Added 'mtp' section to genai_config.json
Saving processing files in model-cpu-new for GenAI
EXIT:0
Fixing tokenizer in model-cpu-new
fixed model-cpu-new/tokenizer.json
fixed model-cpu-new/tokenizer_config.json
Done.