Valid precision + execution provider combinations are: FP32 CPU, FP32 CUDA, FP16 CUDA, FP16 DML, FP16 TRT-RTX, BF16 CUDA, BF16 TRT-RTX, INT8 CPU, INT8 CUDA, INT8 WebGPU, INT4 CPU, INT4 CUDA, INT4 DML, INT4 WebGPU
Extra options: {'block_size': '32', 'algo_config': 'k_quant', 'matmul_mixed_precision': 'last_matmul:int8'}
GroupQueryAttention (GQA) is used in this model.
GroupQueryAttention (GQA) is used in this model.

Loading weights:   0%|          | 0/426 [00:00<?, ?it/s]
Loading weights:  20%|██        | 86/426 [00:00<00:00, 792.49it/s]
Loading weights:  39%|███▉      | 166/426 [00:00<00:00, 775.98it/s]
Loading weights:  57%|█████▋    | 244/426 [00:00<00:00, 691.74it/s]
Loading weights:  74%|███████▎  | 314/426 [00:00<00:00, 669.51it/s]
Loading weights:  90%|████████▉ | 382/426 [00:00<00:00, 652.09it/s]
Loading weights: 100%|██████████| 426/426 [00:00<00:00, 665.99it/s]
2026-11-05 14:07:19,131 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize model with k_quant algorithm...
2026-11-05 14:07:19,142 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_Q8G32' of node '/model/embed_tokens/Reshape' (::Reshape:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-05 14:07:19,142 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_scale' of node '/model/embed_tokens/GatherBlockQuantized' (com.microsoft::GatherBlockQuantized:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-05 14:07:19,142 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_zp' of node '/model/embed_tokens/GatherBlockQuantized' (com.microsoft::GatherBlockQuantized:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-05 14:07:19,151 neural_compressor [WARNING] - Model size > 2GB. Please use model path instead of onnx model object to quantize
2026-11-05 14:07:19,166 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:22,764 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:24,597 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:26,330 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:34,079 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:37,864 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:41,322 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:43,114 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:44,734 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:53,173 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:07:56,999 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:00,504 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:02,221 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:04,025 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:11,691 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:15,492 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:19,762 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:21,361 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:30,188 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:34,028 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:37,552 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:39,298 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:41,104 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:48,691 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:52,485 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:55,910 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:57,566 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:08:59,311 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:08,916 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:12,766 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:16,235 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:18,040 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:19,757 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:27,751 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:31,682 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:35,939 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:37,746 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:46,754 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:50,731 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:54,284 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:56,166 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:09:58,003 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:05,836 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:09,760 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:13,173 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:14,959 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:16,831 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:25,721 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:29,636 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:33,198 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:35,036 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:36,883 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:44,738 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:48,659 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:52,928 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:10:54,787 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:02,853 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:06,749 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:10,337 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:12,124 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:14,010 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:21,895 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:25,844 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:29,291 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:31,135 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:32,910 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:41,075 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:44,978 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:48,590 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:50,449 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:11:52,182 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:00,146 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:03,986 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:08,407 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:10,261 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:18,167 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:22,050 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:25,561 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:27,509 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:29,270 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:37,105 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:40,806 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:44,446 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:46,309 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:48,172 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:12:57,436 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:01,361 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:04,939 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:06,740 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:08,518 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:16,365 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:20,155 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:24,593 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:26,434 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:34,576 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:38,499 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:42,112 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:43,929 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:45,777 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:53,627 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:13:57,400 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:01,001 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:02,893 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:04,729 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:12,699 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:16,646 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:20,168 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:21,983 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:23,818 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:31,507 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:35,427 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:39,864 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:41,684 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:49,589 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:53,510 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:57,037 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:14:58,903 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:00,744 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:08,482 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:12,450 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:16,032 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:17,889 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:19,627 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:27,484 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:31,265 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:34,716 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:36,520 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:38,274 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:45,875 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:49,685 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:53,938 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:15:55,711 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:03,384 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:07,196 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:10,634 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:12,413 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:14,142 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:21,652 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:25,463 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:28,918 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:30,699 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:32,479 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
Reading embedding layer
Reading layer 0
Reading layer 1
Reading layer 2
Reading layer 3
Reading layer 4
Reading layer 5
Reading layer 6
Reading layer 7
Reading layer 8
Reading layer 9
Reading layer 10
Reading layer 11
Reading layer 12
Reading layer 13
Reading layer 14
Reading layer 15
Reading layer 16
Reading layer 17
Reading layer 18
Reading layer 19
Reading layer 20
Reading layer 21
Reading layer 22
Reading layer 23
Reading layer 24
Reading layer 25
Reading layer 26
Reading layer 27
Reading layer 28
Reading layer 29
Reading layer 30
Reading layer 31
Reading final norm
Reading LM head
Building MTP (multi-token prediction) head -> mtp.onnx
Saving ONNX model in model-cuda-int4-slim

Progress: [                    ] 0.50%
Progress: [                    ] 1.00%
Progress: [                    ] 1.49%
Progress: [                    ] 1.99%
Progress: [                    ] 2.49%
Progress: [                    ] 2.99%
Progress: [                    ] 3.48%
Progress: [                    ] 3.98%
Progress: [                    ] 4.48%
Progress: [                    ] 4.98%
Progress: [#                   ] 5.47%
Progress: [#                   ] 5.97%
Progress: [#                   ] 6.47%
Progress: [#                   ] 6.97%
Progress: [#                   ] 7.46%
Progress: [#                   ] 7.96%
Progress: [#                   ] 8.46%
Progress: [#                   ] 8.96%
Progress: [#                   ] 9.45%
Progress: [#                   ] 9.95%
Progress: [##                  ] 10.45%
Progress: [##                  ] 10.95%
Progress: [##                  ] 11.44%
Progress: [##                  ] 11.94%
Progress: [##                  ] 12.44%
Progress: [##                  ] 12.94%
Progress: [##                  ] 13.43%
Progress: [##                  ] 13.93%
Progress: [##                  ] 14.43%
Progress: [##                  ] 14.93%
Progress: [###                 ] 15.42%
Progress: [###                 ] 15.92%
Progress: [###                 ] 16.42%
Progress: [###                 ] 16.92%
Progress: [###                 ] 17.41%
Progress: [###                 ] 17.91%
Progress: [###                 ] 18.41%
Progress: [###                 ] 18.91%
Progress: [###                 ] 19.40%
Progress: [###                 ] 19.90%
Progress: [####                ] 20.40%
Progress: [####                ] 20.90%
Progress: [####                ] 21.39%
Progress: [####                ] 21.89%
Progress: [####                ] 22.39%
Progress: [####                ] 22.89%
Progress: [####                ] 23.38%
Progress: [####                ] 23.88%
Progress: [####                ] 24.38%
Progress: [####                ] 24.88%
Progress: [#####               ] 25.37%
Progress: [#####               ] 25.87%
Progress: [#####               ] 26.37%
Progress: [#####               ] 26.87%
Progress: [#####               ] 27.36%
Progress: [#####               ] 27.86%
Progress: [#####               ] 28.36%
Progress: [#####               ] 28.86%
Progress: [#####               ] 29.35%
Progress: [#####               ] 29.85%
Progress: [######              ] 30.35%
Progress: [######              ] 30.85%
Progress: [######              ] 31.34%
Progress: [######              ] 31.84%
Progress: [######              ] 32.34%
Progress: [######              ] 32.84%
Progress: [######              ] 33.33%
Progress: [######              ] 33.83%
Progress: [######              ] 34.33%
Progress: [######              ] 34.83%
Progress: [#######             ] 35.32%
Progress: [#######             ] 35.82%
Progress: [#######             ] 36.32%
Progress: [#######             ] 36.82%
Progress: [#######             ] 37.31%
Progress: [#######             ] 37.81%
Progress: [#######             ] 38.31%
Progress: [#######             ] 38.81%
Progress: [#######             ] 39.30%
Progress: [#######             ] 39.80%
Progress: [########            ] 40.30%
Progress: [########            ] 40.80%
Progress: [########            ] 41.29%
Progress: [########            ] 41.79%
Progress: [########            ] 42.29%
Progress: [########            ] 42.79%
Progress: [########            ] 43.28%
Progress: [########            ] 43.78%
Progress: [########            ] 44.28%
Progress: [########            ] 44.78%
Progress: [#########           ] 45.27%
Progress: [#########           ] 45.77%
Progress: [#########           ] 46.27%
Progress: [#########           ] 46.77%
Progress: [#########           ] 47.26%
Progress: [#########           ] 47.76%
Progress: [#########           ] 48.26%
Progress: [#########           ] 48.76%
Progress: [#########           ] 49.25%
Progress: [#########           ] 49.75%
Progress: [##########          ] 50.25%
Progress: [##########          ] 50.75%
Progress: [##########          ] 51.24%
Progress: [##########          ] 51.74%
Progress: [##########          ] 52.24%
Progress: [##########          ] 52.74%
Progress: [##########          ] 53.23%
Progress: [##########          ] 53.73%
Progress: [##########          ] 54.23%
Progress: [##########          ] 54.73%
Progress: [###########         ] 55.22%
Progress: [###########         ] 55.72%
Progress: [###########         ] 56.22%
Progress: [###########         ] 56.72%
Progress: [###########         ] 57.21%
Progress: [###########         ] 57.71%
Progress: [###########         ] 58.21%
Progress: [###########         ] 58.71%
Progress: [###########         ] 59.20%
Progress: [###########         ] 59.70%
Progress: [############        ] 60.20%
Progress: [############        ] 60.70%
Progress: [############        ] 61.19%
Progress: [############        ] 61.69%
Progress: [############        ] 62.19%
Progress: [############        ] 62.69%
Progress: [############        ] 63.18%
Progress: [############        ] 63.68%
Progress: [############        ] 64.18%
Progress: [############        ] 64.68%
Progress: [#############       ] 65.17%
Progress: [#############       ] 65.67%
Progress: [#############       ] 66.17%
Progress: [#############       ] 66.67%
Progress: [#############       ] 67.16%
Progress: [#############       ] 67.66%
Progress: [#############       ] 68.16%
Progress: [#############       ] 68.66%
Progress: [#############       ] 69.15%
Progress: [#############       ] 69.65%
Progress: [##############      ] 70.15%
Progress: [##############      ] 70.65%
Progress: [##############      ] 71.14%
Progress: [##############      ] 71.64%
Progress: [##############      ] 72.14%
Progress: [##############      ] 72.64%
Progress: [##############      ] 73.13%
Progress: [##############      ] 73.63%
Progress: [##############      ] 74.13%
Progress: [##############      ] 74.63%
Progress: [###############     ] 75.12%
Progress: [###############     ] 75.62%
Progress: [###############     ] 76.12%
Progress: [###############     ] 76.62%
Progress: [###############     ] 77.11%
Progress: [###############     ] 77.61%
Progress: [###############     ] 78.11%
Progress: [###############     ] 78.61%
Progress: [###############     ] 79.10%
Progress: [###############     ] 79.60%
Progress: [################    ] 80.10%
Progress: [################    ] 80.60%
Progress: [################    ] 81.09%
Progress: [################    ] 81.59%
Progress: [################    ] 82.09%
Progress: [################    ] 82.59%
Progress: [################    ] 83.08%
Progress: [################    ] 83.58%
Progress: [################    ] 84.08%
Progress: [################    ] 84.58%
Progress: [#################   ] 85.07%
Progress: [#################   ] 85.57%
Progress: [#################   ] 86.07%
Progress: [#################   ] 86.57%
Progress: [#################   ] 87.06%
Progress: [#################   ] 87.56%
Progress: [#################   ] 88.06%
Progress: [#################   ] 88.56%
Progress: [#################   ] 89.05%
Progress: [#################   ] 89.55%
Progress: [##################  ] 90.05%
Progress: [##################  ] 90.55%
Progress: [##################  ] 91.04%
Progress: [##################  ] 91.54%
Progress: [##################  ] 92.04%
Progress: [##################  ] 92.54%
Progress: [##################  ] 93.03%
Progress: [##################  ] 93.53%
Progress: [##################  ] 94.03%2026-11-05 14:16:40,092 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:43,875 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:47,369 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:49,140 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:50,845 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:16:58,339 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:17:02,168 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:17:06,472 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:17:08,285 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:17:15,949 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:17:20,056 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:19:54,455 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of model with k_quant algorithm.

Progress: [##################  ] 94.53%
Progress: [################### ] 95.02%
Progress: [################### ] 95.52%
Progress: [################### ] 96.02%
Progress: [################### ] 96.52%
Progress: [################### ] 97.01%
Progress: [################### ] 97.51%
Progress: [################### ] 98.01%
Progress: [################### ] 98.51%
Progress: [################### ] 99.00%
Progress: [################### ] 99.50%
Progress: [####################] 100.00%
0it [00:00, ?it/s]
Saving model.layers.0.input_layernorm.weight (f16, [2560]):   0%|          | 1/710 [00:00<00:11, 61.35it/s]
Saving model.layers.0.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):   0%|          | 2/710 [00:00<00:05, 121.94it/s]
Saving model.layers.0.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):   0%|          | 3/710 [00:00<00:03, 181.88it/s]
Saving model.layers.0.linear_attn.conv1d.weight (f16, [8192,1,4]):   1%|          | 4/710 [00:00<00:02, 240.60it/s]      
Saving model.layers.0.linear_attn.conv1d.bias (f16, [8192]):   1%|          | 5/710 [00:00<00:02, 299.75it/s]      
Saving model.layers.0.linear_attn.A_log (f32, [32]):   1%|          | 6/710 [00:00<00:01, 358.66it/s]        
Saving model.layers.0.linear_attn.dt_bias (f32, [32]):   1%|          | 7/710 [00:00<00:01, 417.73it/s]
Saving model.layers.0.linear_attn.norm.weight (f16, [128]):   1%|          | 8/710 [00:00<00:01, 476.62it/s]
Saving model.layers.0.post_attention_layernorm.weight (f16, [2560]):   1%|▏         | 9/710 [00:00<00:01, 535.38it/s]
Saving model.layers.1.input_layernorm.weight (f16, [2560]):   1%|▏         | 10/710 [00:00<00:01, 593.23it/s]        
Saving model.layers.1.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):   2%|▏         | 11/710 [00:00<00:01, 651.32it/s]
Saving model.layers.1.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):   2%|▏         | 12/710 [00:00<00:00, 700.44it/s]
Saving model.layers.1.linear_attn.conv1d.weight (f16, [8192,1,4]):   2%|▏         | 13/710 [00:00<00:00, 756.01it/s]      
Saving model.layers.1.linear_attn.conv1d.bias (f16, [8192]):   2%|▏         | 14/710 [00:00<00:00, 807.13it/s]      
Saving model.layers.1.linear_attn.A_log (f32, [32]):   2%|▏         | 15/710 [00:00<00:00, 862.64it/s]        
Saving model.layers.1.linear_attn.dt_bias (f32, [32]):   2%|▏         | 16/710 [00:00<00:00, 918.53it/s]
Saving model.layers.1.linear_attn.norm.weight (f16, [128]):   2%|▏         | 17/710 [00:00<00:00, 974.53it/s]
Saving model.layers.1.post_attention_layernorm.weight (f16, [2560]):   3%|▎         | 18/710 [00:00<00:00, 1030.43it/s]
Saving model.layers.2.input_layernorm.weight (f16, [2560]):   3%|▎         | 19/710 [00:00<00:00, 1085.50it/s]         
Saving model.layers.2.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):   3%|▎         | 20/710 [00:00<00:00, 1140.61it/s]
Saving model.layers.2.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):   3%|▎         | 21/710 [00:00<00:00, 1193.76it/s]
Saving model.layers.2.linear_attn.conv1d.weight (f16, [8192,1,4]):   3%|▎         | 22/710 [00:00<00:00, 1246.16it/s]      
Saving model.layers.2.linear_attn.conv1d.bias (f16, [8192]):   3%|▎         | 23/710 [00:00<00:00, 1299.46it/s]      
Saving model.layers.2.linear_attn.A_log (f32, [32]):   3%|▎         | 24/710 [00:00<00:00, 1353.40it/s]        
Saving model.layers.2.linear_attn.dt_bias (f32, [32]):   4%|▎         | 25/710 [00:00<00:00, 1407.77it/s]
Saving model.layers.2.linear_attn.norm.weight (f16, [128]):   4%|▎         | 26/710 [00:00<00:00, 1462.12it/s]
Saving model.layers.2.post_attention_layernorm.weight (f16, [2560]):   4%|▍         | 27/710 [00:00<00:00, 1516.34it/s]
Saving model.layers.3.input_layernorm.weight (f16, [2560]):   4%|▍         | 28/710 [00:00<00:00, 1570.00it/s]         
Saving model.layers.3.attn.q_norm.layernorm.weight (f16, [256]):   4%|▍         | 29/710 [00:00<00:00, 1623.44it/s]
Saving model.layers.3.attn.k_norm.layernorm.weight (f16, [256]):   4%|▍         | 30/710 [00:00<00:00, 1677.32it/s]
Saving cos_cache (f32, [262144,32]):   4%|▍         | 31/710 [00:00<00:00, 1730.81it/s]                            
Saving sin_cache (f32, [262144,32]):   5%|▍         | 32/710 [00:00<00:00, 1051.57it/s]
Saving model.layers.3.post_attention_layernorm.weight (f16, [2560]):   5%|▍         | 33/710 [00:00<00:00, 855.35it/s]
Saving model.layers.4.input_layernorm.weight (f16, [2560]):   5%|▍         | 34/710 [00:00<00:00, 877.99it/s]         
Saving model.layers.4.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):   5%|▍         | 35/710 [00:00<00:00, 902.72it/s]
Saving model.layers.4.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):   5%|▌         | 36/710 [00:00<00:00, 922.95it/s]
Saving model.layers.4.linear_attn.conv1d.weight (f16, [8192,1,4]):   5%|▌         | 37/710 [00:00<00:00, 947.05it/s]      
Saving model.layers.4.linear_attn.conv1d.bias (f16, [8192]):   5%|▌         | 38/710 [00:00<00:00, 969.78it/s]      
Saving model.layers.4.linear_attn.A_log (f32, [32]):   5%|▌         | 39/710 [00:00<00:00, 994.22it/s]        
Saving model.layers.4.linear_attn.dt_bias (f32, [32]):   6%|▌         | 40/710 [00:00<00:00, 1018.22it/s]
Saving model.layers.4.linear_attn.norm.weight (f16, [128]):   6%|▌         | 41/710 [00:00<00:00, 1042.00it/s]
Saving model.layers.4.post_attention_layernorm.weight (f16, [2560]):   6%|▌         | 42/710 [00:00<00:00, 1066.37it/s]
Saving model.layers.5.input_layernorm.weight (f16, [2560]):   6%|▌         | 43/710 [00:00<00:00, 1090.49it/s]         
Saving model.layers.5.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):   6%|▌         | 44/710 [00:00<00:00, 1114.75it/s]
Saving model.layers.5.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):   6%|▋         | 45/710 [00:00<00:00, 1137.83it/s]
Saving model.layers.5.linear_attn.conv1d.weight (f16, [8192,1,4]):   6%|▋         | 46/710 [00:00<00:00, 1160.52it/s]      
Saving model.layers.5.linear_attn.conv1d.bias (f16, [8192]):   7%|▋         | 47/710 [00:00<00:00, 1180.18it/s]      
Saving model.layers.5.linear_attn.A_log (f32, [32]):   7%|▋         | 48/710 [00:00<00:00, 1203.90it/s]        
Saving model.layers.5.linear_attn.dt_bias (f32, [32]):   7%|▋         | 49/710 [00:00<00:00, 1228.08it/s]
Saving model.layers.5.linear_attn.norm.weight (f16, [128]):   7%|▋         | 50/710 [00:00<00:00, 1252.26it/s]
Saving model.layers.5.post_attention_layernorm.weight (f16, [2560]):   7%|▋         | 51/710 [00:00<00:00, 1276.46it/s]
Saving model.layers.6.input_layernorm.weight (f16, [2560]):   7%|▋         | 52/710 [00:00<00:00, 1299.93it/s]         
Saving model.layers.6.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):   7%|▋         | 53/710 [00:00<00:00, 1323.73it/s]
Saving model.layers.6.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):   8%|▊         | 54/710 [00:00<00:00, 1346.49it/s]
Saving model.layers.6.linear_attn.conv1d.weight (f16, [8192,1,4]):   8%|▊         | 55/710 [00:00<00:00, 1365.58it/s]      
Saving model.layers.6.linear_attn.conv1d.bias (f16, [8192]):   8%|▊         | 56/710 [00:00<00:00, 1388.05it/s]      
Saving model.layers.6.linear_attn.A_log (f32, [32]):   8%|▊         | 57/710 [00:00<00:00, 1411.31it/s]        
Saving model.layers.6.linear_attn.dt_bias (f32, [32]):   8%|▊         | 58/710 [00:00<00:00, 1435.06it/s]
Saving model.layers.6.linear_attn.norm.weight (f16, [128]):   8%|▊         | 59/710 [00:00<00:00, 1458.81it/s]
Saving model.layers.6.post_attention_layernorm.weight (f16, [2560]):   8%|▊         | 60/710 [00:00<00:00, 1482.64it/s]
Saving model.layers.7.input_layernorm.weight (f16, [2560]):   9%|▊         | 61/710 [00:00<00:00, 1506.21it/s]         
Saving model.layers.7.attn.q_norm.layernorm.weight (f16, [256]):   9%|▊         | 62/710 [00:00<00:00, 1529.81it/s]
Saving model.layers.7.attn.k_norm.layernorm.weight (f16, [256]):   9%|▉         | 63/710 [00:00<00:00, 1553.59it/s]
Saving model.layers.7.post_attention_layernorm.weight (f16, [2560]):   9%|▉         | 64/710 [00:00<00:00, 1577.26it/s]
Saving model.layers.8.input_layernorm.weight (f16, [2560]):   9%|▉         | 65/710 [00:00<00:00, 1600.68it/s]         
Saving model.layers.8.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):   9%|▉         | 66/710 [00:00<00:00, 1624.17it/s]
Saving model.layers.8.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):   9%|▉         | 67/710 [00:00<00:00, 1645.89it/s]
Saving model.layers.8.linear_attn.conv1d.weight (f16, [8192,1,4]):  10%|▉         | 68/710 [00:00<00:00, 1664.06it/s]      
Saving model.layers.8.linear_attn.conv1d.bias (f16, [8192]):  10%|▉         | 69/710 [00:00<00:00, 1686.28it/s]      
Saving model.layers.8.linear_attn.A_log (f32, [32]):  10%|▉         | 70/710 [00:00<00:00, 1709.28it/s]        
Saving model.layers.8.linear_attn.dt_bias (f32, [32]):  10%|█         | 71/710 [00:00<00:00, 1732.75it/s]
Saving model.layers.8.linear_attn.norm.weight (f16, [128]):  10%|█         | 72/710 [00:00<00:00, 1756.00it/s]
Saving model.layers.8.post_attention_layernorm.weight (f16, [2560]):  10%|█         | 73/710 [00:00<00:00, 1779.31it/s]
Saving model.layers.9.input_layernorm.weight (f16, [2560]):  10%|█         | 74/710 [00:00<00:00, 1802.40it/s]         
Saving model.layers.9.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  11%|█         | 75/710 [00:00<00:00, 1825.34it/s]
Saving model.layers.9.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  11%|█         | 76/710 [00:00<00:00, 1846.23it/s]
Saving model.layers.9.linear_attn.conv1d.weight (f16, [8192,1,4]):  11%|█         | 77/710 [00:00<00:00, 1867.65it/s]      
Saving model.layers.9.linear_attn.conv1d.bias (f16, [8192]):  11%|█         | 78/710 [00:00<00:00, 1889.84it/s]      
Saving model.layers.9.linear_attn.A_log (f32, [32]):  11%|█         | 79/710 [00:00<00:00, 1912.54it/s]        
Saving model.layers.9.linear_attn.dt_bias (f32, [32]):  11%|█▏        | 80/710 [00:00<00:00, 1935.64it/s]
Saving model.layers.9.linear_attn.norm.weight (f16, [128]):  11%|█▏        | 81/710 [00:00<00:00, 1958.61it/s]
Saving model.layers.9.post_attention_layernorm.weight (f16, [2560]):  12%|█▏        | 82/710 [00:00<00:00, 1981.64it/s]
Saving model.layers.10.input_layernorm.weight (f16, [2560]):  12%|█▏        | 83/710 [00:00<00:00, 2004.26it/s]        
Saving model.layers.10.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  12%|█▏        | 84/710 [00:00<00:00, 2018.75it/s]
Saving model.layers.10.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  12%|█▏        | 85/710 [00:00<00:00, 2037.63it/s]
Saving model.layers.10.linear_attn.conv1d.weight (f16, [8192,1,4]):  12%|█▏        | 86/710 [00:00<00:00, 2057.30it/s]      
Saving model.layers.10.linear_attn.conv1d.bias (f16, [8192]):  12%|█▏        | 87/710 [00:00<00:00, 2078.36it/s]      
Saving model.layers.10.linear_attn.A_log (f32, [32]):  12%|█▏        | 88/710 [00:00<00:00, 2100.24it/s]        
Saving model.layers.10.linear_attn.dt_bias (f32, [32]):  13%|█▎        | 89/710 [00:00<00:00, 2122.83it/s]
Saving model.layers.10.linear_attn.norm.weight (f16, [128]):  13%|█▎        | 90/710 [00:00<00:00, 2145.30it/s]
Saving model.layers.10.post_attention_layernorm.weight (f16, [2560]):  13%|█▎        | 91/710 [00:00<00:00, 2167.83it/s]
Saving model.layers.11.input_layernorm.weight (f16, [2560]):  13%|█▎        | 92/710 [00:00<00:00, 2189.93it/s]         
Saving model.layers.11.attn.q_norm.layernorm.weight (f16, [256]):  13%|█▎        | 93/710 [00:00<00:00, 2212.09it/s]
Saving model.layers.11.attn.k_norm.layernorm.weight (f16, [256]):  13%|█▎        | 94/710 [00:00<00:00, 2234.64it/s]
Saving model.layers.11.post_attention_layernorm.weight (f16, [2560]):  13%|█▎        | 95/710 [00:00<00:00, 2257.05it/s]
Saving model.layers.12.input_layernorm.weight (f16, [2560]):  14%|█▎        | 96/710 [00:00<00:00, 2279.19it/s]         
Saving model.layers.12.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  14%|█▎        | 97/710 [00:00<00:00, 2301.30it/s]
Saving model.layers.12.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  14%|█▍        | 98/710 [00:00<00:00, 2321.67it/s]
Saving model.layers.12.linear_attn.conv1d.weight (f16, [8192,1,4]):  14%|█▍        | 99/710 [00:00<00:00, 2341.83it/s]      
Saving model.layers.12.linear_attn.conv1d.bias (f16, [8192]):  14%|█▍        | 100/710 [00:00<00:00, 2354.29it/s]     
Saving model.layers.12.linear_attn.A_log (f32, [32]):  14%|█▍        | 101/710 [00:00<00:00, 2375.11it/s]        
Saving model.layers.12.linear_attn.dt_bias (f32, [32]):  14%|█▍        | 102/710 [00:00<00:00, 2397.15it/s]
Saving model.layers.12.linear_attn.norm.weight (f16, [128]):  15%|█▍        | 103/710 [00:00<00:00, 2418.89it/s]
Saving model.layers.12.post_attention_layernorm.weight (f16, [2560]):  15%|█▍        | 104/710 [00:00<00:00, 2440.76it/s]
Saving model.layers.13.input_layernorm.weight (f16, [2560]):  15%|█▍        | 105/710 [00:00<00:00, 2462.34it/s]         
Saving model.layers.13.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  15%|█▍        | 106/710 [00:00<00:00, 2483.87it/s]
Saving model.layers.13.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  15%|█▌        | 107/710 [00:00<00:00, 2503.34it/s]
Saving model.layers.13.linear_attn.conv1d.weight (f16, [8192,1,4]):  15%|█▌        | 108/710 [00:00<00:00, 2522.36it/s]      
Saving model.layers.13.linear_attn.conv1d.bias (f16, [8192]):  15%|█▌        | 109/710 [00:00<00:00, 2542.13it/s]      
Saving model.layers.13.linear_attn.A_log (f32, [32]):  15%|█▌        | 110/710 [00:00<00:00, 2562.26it/s]        
Saving model.layers.13.linear_attn.dt_bias (f32, [32]):  16%|█▌        | 111/710 [00:00<00:00, 2584.03it/s]
Saving model.layers.13.linear_attn.norm.weight (f16, [128]):  16%|█▌        | 112/710 [00:00<00:00, 2605.68it/s]
Saving model.layers.13.post_attention_layernorm.weight (f16, [2560]):  16%|█▌        | 113/710 [00:00<00:00, 2627.37it/s]
Saving model.layers.14.input_layernorm.weight (f16, [2560]):  16%|█▌        | 114/710 [00:00<00:00, 2646.37it/s]         
Saving model.layers.14.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  16%|█▌        | 115/710 [00:00<00:00, 2666.78it/s]
Saving model.layers.14.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  16%|█▋        | 116/710 [00:00<00:00, 2684.81it/s]
Saving model.layers.14.linear_attn.conv1d.weight (f16, [8192,1,4]):  16%|█▋        | 117/710 [00:00<00:00, 2703.24it/s]      
Saving model.layers.14.linear_attn.conv1d.bias (f16, [8192]):  17%|█▋        | 118/710 [00:00<00:00, 2723.14it/s]      
Saving model.layers.14.linear_attn.A_log (f32, [32]):  17%|█▋        | 119/710 [00:00<00:00, 2743.97it/s]        
Saving model.layers.14.linear_attn.dt_bias (f32, [32]):  17%|█▋        | 120/710 [00:00<00:00, 2765.55it/s]
Saving model.layers.14.linear_attn.norm.weight (f16, [128]):  17%|█▋        | 121/710 [00:00<00:00, 2786.55it/s]
Saving model.layers.14.post_attention_layernorm.weight (f16, [2560]):  17%|█▋        | 122/710 [00:00<00:00, 2807.86it/s]
Saving model.layers.15.input_layernorm.weight (f16, [2560]):  17%|█▋        | 123/710 [00:00<00:00, 2828.86it/s]         
Saving model.layers.15.attn.q_norm.layernorm.weight (f16, [256]):  17%|█▋        | 124/710 [00:00<00:00, 2848.14it/s]
Saving model.layers.15.attn.k_norm.layernorm.weight (f16, [256]):  18%|█▊        | 125/710 [00:00<00:00, 2869.46it/s]
Saving model.layers.15.post_attention_layernorm.weight (f16, [2560]):  18%|█▊        | 126/710 [00:00<00:00, 2890.69it/s]
Saving model.layers.16.input_layernorm.weight (f16, [2560]):  18%|█▊        | 127/710 [00:00<00:00, 2910.63it/s]         
Saving model.layers.16.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  18%|█▊        | 128/710 [00:00<00:00, 2931.56it/s]
Saving model.layers.16.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  18%|█▊        | 129/710 [00:00<00:00, 2949.65it/s]
Saving model.layers.16.linear_attn.conv1d.weight (f16, [8192,1,4]):  18%|█▊        | 130/710 [00:00<00:00, 2967.90it/s]      
Saving model.layers.16.linear_attn.conv1d.bias (f16, [8192]):  18%|█▊        | 131/710 [00:00<00:00, 2983.91it/s]      
Saving model.layers.16.linear_attn.A_log (f32, [32]):  19%|█▊        | 132/710 [00:00<00:00, 3004.08it/s]        
Saving model.layers.16.linear_attn.dt_bias (f32, [32]):  19%|█▊        | 133/710 [00:00<00:00, 3025.31it/s]
Saving model.layers.16.linear_attn.norm.weight (f16, [128]):  19%|█▉        | 134/710 [00:00<00:00, 3046.32it/s]
Saving model.layers.16.post_attention_layernorm.weight (f16, [2560]):  19%|█▉        | 135/710 [00:00<00:00, 3067.29it/s]
Saving model.layers.17.input_layernorm.weight (f16, [2560]):  19%|█▉        | 136/710 [00:00<00:00, 3087.80it/s]         
Saving model.layers.17.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  19%|█▉        | 137/710 [00:00<00:00, 3108.51it/s]
Saving model.layers.17.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  19%|█▉        | 138/710 [00:00<00:00, 3126.09it/s]
Saving model.layers.17.linear_attn.conv1d.weight (f16, [8192,1,4]):  20%|█▉        | 139/710 [00:00<00:00, 3144.33it/s]      
Saving model.layers.17.linear_attn.conv1d.bias (f16, [8192]):  20%|█▉        | 140/710 [00:00<00:00, 3163.50it/s]      
Saving model.layers.17.linear_attn.A_log (f32, [32]):  20%|█▉        | 141/710 [00:00<00:00, 3183.73it/s]        
Saving model.layers.17.linear_attn.dt_bias (f32, [32]):  20%|██        | 142/710 [00:00<00:00, 3204.57it/s]
Saving model.layers.17.linear_attn.norm.weight (f16, [128]):  20%|██        | 143/710 [00:00<00:00, 3224.93it/s]
Saving model.layers.17.post_attention_layernorm.weight (f16, [2560]):  20%|██        | 144/710 [00:00<00:00, 3245.72it/s]
Saving model.layers.18.input_layernorm.weight (f16, [2560]):  20%|██        | 145/710 [00:00<00:00, 3266.08it/s]         
Saving model.layers.18.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  21%|██        | 146/710 [00:00<00:00, 3285.98it/s]
Saving model.layers.18.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  21%|██        | 147/710 [00:00<00:00, 3304.53it/s]
Saving model.layers.18.linear_attn.conv1d.weight (f16, [8192,1,4]):  21%|██        | 148/710 [00:00<00:00, 3322.75it/s]      
Saving model.layers.18.linear_attn.conv1d.bias (f16, [8192]):  21%|██        | 149/710 [00:00<00:00, 3332.72it/s]      
Saving model.layers.18.linear_attn.A_log (f32, [32]):  21%|██        | 150/710 [00:00<00:00, 3352.58it/s]        
Saving model.layers.18.linear_attn.dt_bias (f32, [32]):  21%|██▏       | 151/710 [00:00<00:00, 3373.24it/s]
Saving model.layers.18.linear_attn.norm.weight (f16, [128]):  21%|██▏       | 152/710 [00:00<00:00, 3393.74it/s]
Saving model.layers.18.post_attention_layernorm.weight (f16, [2560]):  22%|██▏       | 153/710 [00:00<00:00, 3414.10it/s]
Saving model.layers.19.input_layernorm.weight (f16, [2560]):  22%|██▏       | 154/710 [00:00<00:00, 3433.22it/s]         
Saving model.layers.19.attn.q_norm.layernorm.weight (f16, [256]):  22%|██▏       | 155/710 [00:00<00:00, 3452.47it/s]
Saving model.layers.19.attn.k_norm.layernorm.weight (f16, [256]):  22%|██▏       | 156/710 [00:00<00:00, 3472.22it/s]
Saving model.layers.19.post_attention_layernorm.weight (f16, [2560]):  22%|██▏       | 157/710 [00:00<00:00, 3492.27it/s]
Saving model.layers.20.input_layernorm.weight (f16, [2560]):  22%|██▏       | 158/710 [00:00<00:00, 3511.66it/s]         
Saving model.layers.20.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  22%|██▏       | 159/710 [00:00<00:00, 3531.21it/s]
Saving model.layers.20.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  23%|██▎       | 160/710 [00:00<00:00, 3548.27it/s]
Saving model.layers.20.linear_attn.conv1d.weight (f16, [8192,1,4]):  23%|██▎       | 161/710 [00:00<00:00, 3565.40it/s]      
Saving model.layers.20.linear_attn.conv1d.bias (f16, [8192]):  23%|██▎       | 162/710 [00:00<00:00, 3580.98it/s]      
Saving model.layers.20.linear_attn.A_log (f32, [32]):  23%|██▎       | 163/710 [00:00<00:00, 3599.64it/s]        
Saving model.layers.20.linear_attn.dt_bias (f32, [32]):  23%|██▎       | 164/710 [00:00<00:00, 3619.83it/s]
Saving model.layers.20.linear_attn.norm.weight (f16, [128]):  23%|██▎       | 165/710 [00:00<00:00, 3639.84it/s]
Saving model.layers.20.post_attention_layernorm.weight (f16, [2560]):  23%|██▎       | 166/710 [00:00<00:00, 3659.95it/s]
Saving model.layers.21.input_layernorm.weight (f16, [2560]):  24%|██▎       | 167/710 [00:00<00:00, 3679.56it/s]         
Saving model.layers.21.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  24%|██▎       | 168/710 [00:00<00:00, 3698.68it/s]
Saving model.layers.21.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  24%|██▍       | 169/710 [00:00<00:00, 3716.37it/s]
Saving model.layers.21.linear_attn.conv1d.weight (f16, [8192,1,4]):  24%|██▍       | 170/710 [00:00<00:00, 3733.38it/s]      
Saving model.layers.21.linear_attn.conv1d.bias (f16, [8192]):  24%|██▍       | 171/710 [00:00<00:00, 3751.61it/s]      
Saving model.layers.21.linear_attn.A_log (f32, [32]):  24%|██▍       | 172/710 [00:00<00:00, 3770.20it/s]        
Saving model.layers.21.linear_attn.dt_bias (f32, [32]):  24%|██▍       | 173/710 [00:00<00:00, 3790.26it/s]
Saving model.layers.21.linear_attn.norm.weight (f16, [128]):  25%|██▍       | 174/710 [00:00<00:00, 3810.08it/s]
Saving model.layers.21.post_attention_layernorm.weight (f16, [2560]):  25%|██▍       | 175/710 [00:00<00:00, 3829.64it/s]
Saving model.layers.22.input_layernorm.weight (f16, [2560]):  25%|██▍       | 176/710 [00:00<00:00, 3848.69it/s]         
Saving model.layers.22.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  25%|██▍       | 177/710 [00:00<00:00, 3868.16it/s]
Saving model.layers.22.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  25%|██▌       | 178/710 [00:00<00:00, 3885.21it/s]
Saving model.layers.22.linear_attn.conv1d.weight (f16, [8192,1,4]):  25%|██▌       | 179/710 [00:00<00:00, 3901.62it/s]      
Saving model.layers.22.linear_attn.conv1d.bias (f16, [8192]):  25%|██▌       | 180/710 [00:00<00:00, 3909.64it/s]      
Saving model.layers.22.linear_attn.A_log (f32, [32]):  25%|██▌       | 181/710 [00:00<00:00, 3927.49it/s]        
Saving model.layers.22.linear_attn.dt_bias (f32, [32]):  26%|██▌       | 182/710 [00:00<00:00, 3947.21it/s]
Saving model.layers.22.linear_attn.norm.weight (f16, [128]):  26%|██▌       | 183/710 [00:00<00:00, 3966.67it/s]
Saving model.layers.22.post_attention_layernorm.weight (f16, [2560]):  26%|██▌       | 184/710 [00:00<00:00, 3986.28it/s]
Saving model.layers.23.input_layernorm.weight (f16, [2560]):  26%|██▌       | 185/710 [00:00<00:00, 4005.46it/s]         
Saving model.layers.23.attn.q_norm.layernorm.weight (f16, [256]):  26%|██▌       | 186/710 [00:00<00:00, 4024.43it/s]
Saving model.layers.23.attn.k_norm.layernorm.weight (f16, [256]):  26%|██▋       | 187/710 [00:00<00:00, 4043.94it/s]
Saving model.layers.23.post_attention_layernorm.weight (f16, [2560]):  26%|██▋       | 188/710 [00:00<00:00, 4063.31it/s]
Saving model.layers.24.input_layernorm.weight (f16, [2560]):  27%|██▋       | 189/710 [00:00<00:00, 4082.33it/s]         
Saving model.layers.24.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  27%|██▋       | 190/710 [00:00<00:00, 4101.44it/s]
Saving model.layers.24.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  27%|██▋       | 191/710 [00:00<00:00, 4117.94it/s]
Saving model.layers.24.linear_attn.conv1d.weight (f16, [8192,1,4]):  27%|██▋       | 192/710 [00:00<00:00, 4132.87it/s]      
Saving model.layers.24.linear_attn.conv1d.bias (f16, [8192]):  27%|██▋       | 193/710 [00:00<00:00, 4150.31it/s]      
Saving model.layers.24.linear_attn.A_log (f32, [32]):  27%|██▋       | 194/710 [00:00<00:00, 4166.77it/s]        
Saving model.layers.24.linear_attn.dt_bias (f32, [32]):  27%|██▋       | 195/710 [00:00<00:00, 4186.02it/s]
Saving model.layers.24.linear_attn.norm.weight (f16, [128]):  28%|██▊       | 196/710 [00:00<00:00, 4205.18it/s]
Saving model.layers.24.post_attention_layernorm.weight (f16, [2560]):  28%|██▊       | 197/710 [00:00<00:00, 4223.46it/s]
Saving model.layers.25.input_layernorm.weight (f16, [2560]):  28%|██▊       | 198/710 [00:00<00:00, 4242.04it/s]         
Saving model.layers.25.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  28%|██▊       | 199/710 [00:00<00:00, 4260.76it/s]
Saving model.layers.25.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  28%|██▊       | 200/710 [00:00<00:00, 4276.59it/s]
Saving model.layers.25.linear_attn.conv1d.weight (f16, [8192,1,4]):  28%|██▊       | 201/710 [00:00<00:00, 4292.04it/s]      
Saving model.layers.25.linear_attn.conv1d.bias (f16, [8192]):  28%|██▊       | 202/710 [00:00<00:00, 4305.33it/s]      
Saving model.layers.25.linear_attn.A_log (f32, [32]):  29%|██▊       | 203/710 [00:00<00:00, 4323.37it/s]        
Saving model.layers.25.linear_attn.dt_bias (f32, [32]):  29%|██▊       | 204/710 [00:00<00:00, 4342.63it/s]
Saving model.layers.25.linear_attn.norm.weight (f16, [128]):  29%|██▉       | 205/710 [00:00<00:00, 4361.62it/s]
Saving model.layers.25.post_attention_layernorm.weight (f16, [2560]):  29%|██▉       | 206/710 [00:00<00:00, 4380.70it/s]
Saving model.layers.26.input_layernorm.weight (f16, [2560]):  29%|██▉       | 207/710 [00:00<00:00, 4399.04it/s]         
Saving model.layers.26.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  29%|██▉       | 208/710 [00:00<00:00, 4417.45it/s]
Saving model.layers.26.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  29%|██▉       | 209/710 [00:00<00:00, 4432.67it/s]
Saving model.layers.26.linear_attn.conv1d.weight (f16, [8192,1,4]):  30%|██▉       | 210/710 [00:00<00:00, 4448.10it/s]      
Saving model.layers.26.linear_attn.conv1d.bias (f16, [8192]):  30%|██▉       | 211/710 [00:00<00:00, 4460.23it/s]      
Saving model.layers.26.linear_attn.A_log (f32, [32]):  30%|██▉       | 212/710 [00:00<00:00, 4477.91it/s]        
Saving model.layers.26.linear_attn.dt_bias (f32, [32]):  30%|███       | 213/710 [00:00<00:00, 4496.91it/s]
Saving model.layers.26.linear_attn.norm.weight (f16, [128]):  30%|███       | 214/710 [00:00<00:00, 4515.61it/s]
Saving model.layers.26.post_attention_layernorm.weight (f16, [2560]):  30%|███       | 215/710 [00:00<00:00, 4533.56it/s]
Saving model.layers.27.input_layernorm.weight (f16, [2560]):  30%|███       | 216/710 [00:00<00:00, 4551.74it/s]         
Saving model.layers.27.attn.q_norm.layernorm.weight (f16, [256]):  31%|███       | 217/710 [00:00<00:00, 4570.01it/s]
Saving model.layers.27.attn.k_norm.layernorm.weight (f16, [256]):  31%|███       | 218/710 [00:00<00:00, 4588.84it/s]
Saving model.layers.27.post_attention_layernorm.weight (f16, [2560]):  31%|███       | 219/710 [00:00<00:00, 4607.37it/s]
Saving model.layers.28.input_layernorm.weight (f16, [2560]):  31%|███       | 220/710 [00:00<00:00, 4625.53it/s]         
Saving model.layers.28.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  31%|███       | 221/710 [00:00<00:00, 4643.85it/s]
Saving model.layers.28.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  31%|███▏      | 222/710 [00:00<00:00, 4658.82it/s]
Saving model.layers.28.linear_attn.conv1d.weight (f16, [8192,1,4]):  31%|███▏      | 223/710 [00:00<00:00, 4672.91it/s]      
Saving model.layers.28.linear_attn.conv1d.bias (f16, [8192]):  32%|███▏      | 224/710 [00:00<00:00, 4689.11it/s]      
Saving model.layers.28.linear_attn.A_log (f32, [32]):  32%|███▏      | 225/710 [00:00<00:00, 4705.84it/s]        
Saving model.layers.28.linear_attn.dt_bias (f32, [32]):  32%|███▏      | 226/710 [00:00<00:00, 4724.52it/s]
Saving model.layers.28.linear_attn.norm.weight (f16, [128]):  32%|███▏      | 227/710 [00:00<00:00, 4742.99it/s]
Saving model.layers.28.post_attention_layernorm.weight (f16, [2560]):  32%|███▏      | 228/710 [00:00<00:00, 4761.51it/s]
Saving model.layers.29.input_layernorm.weight (f16, [2560]):  32%|███▏      | 229/710 [00:00<00:00, 4779.23it/s]         
Saving model.layers.29.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  32%|███▏      | 230/710 [00:00<00:00, 4797.35it/s]
Saving model.layers.29.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  33%|███▎      | 231/710 [00:00<00:00, 4812.56it/s]
Saving model.layers.29.linear_attn.conv1d.weight (f16, [8192,1,4]):  33%|███▎      | 232/710 [00:00<00:00, 4827.69it/s]      
Saving model.layers.29.linear_attn.conv1d.bias (f16, [8192]):  33%|███▎      | 233/710 [00:00<00:00, 4831.45it/s]      
Saving model.layers.29.linear_attn.A_log (f32, [32]):  33%|███▎      | 234/710 [00:00<00:00, 4847.90it/s]        
Saving model.layers.29.linear_attn.dt_bias (f32, [32]):  33%|███▎      | 235/710 [00:00<00:00, 4866.36it/s]
Saving model.layers.29.linear_attn.norm.weight (f16, [128]):  33%|███▎      | 236/710 [00:00<00:00, 4884.51it/s]
Saving model.layers.29.post_attention_layernorm.weight (f16, [2560]):  33%|███▎      | 237/710 [00:00<00:00, 4902.81it/s]
Saving model.layers.30.input_layernorm.weight (f16, [2560]):  34%|███▎      | 238/710 [00:00<00:00, 4919.06it/s]         
Saving model.layers.30.linear_attn.b_proj.MatMul.weight (f16, [2560,32]):  34%|███▎      | 239/710 [00:00<00:00, 4936.13it/s]
Saving model.layers.30.linear_attn.a_proj.MatMul.weight (f16, [2560,32]):  34%|███▍      | 240/710 [00:00<00:00, 4950.76it/s]
Saving model.layers.30.linear_attn.conv1d.weight (f16, [8192,1,4]):  34%|███▍      | 241/710 [00:00<00:00, 4963.92it/s]      
Saving model.layers.30.linear_attn.conv1d.bias (f16, [8192]):  34%|███▍      | 242/710 [00:00<00:00, 4973.23it/s]      
Saving model.layers.30.linear_attn.A_log (f32, [32]):  34%|███▍      | 243/710 [00:00<00:00, 4990.11it/s]        
Saving model.layers.30.linear_attn.dt_bias (f32, [32]):  34%|███▍      | 244/710 [00:00<00:00, 5008.39it/s]
Saving model.layers.30.linear_attn.norm.weight (f16, [128]):  35%|███▍      | 245/710 [00:00<00:00, 5026.29it/s]
Saving model.layers.30.post_attention_layernorm.weight (f16, [2560]):  35%|███▍      | 246/710 [00:00<00:00, 5044.41it/s]
Saving model.layers.31.input_layernorm.weight (f16, [2560]):  35%|███▍      | 247/710 [00:00<00:00, 5061.85it/s]         
Saving model.layers.31.attn.q_norm.layernorm.weight (f16, [256]):  35%|███▍      | 248/710 [00:00<00:00, 5079.31it/s]
Saving model.layers.31.attn.k_norm.layernorm.weight (f16, [256]):  35%|███▌      | 249/710 [00:00<00:00, 5097.38it/s]
Saving model.layers.31.post_attention_layernorm.weight (f16, [2560]):  35%|███▌      | 250/710 [00:00<00:00, 5115.18it/s]
Saving model.layers.32.final_norm_layernorm.weight (f16, [2560]):  35%|███▌      | 251/710 [00:00<00:00, 5132.66it/s]    
Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  35%|███▌      | 252/710 [00:00<00:00, 5150.22it/s]
Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  36%|███▌      | 253/710 [00:00<00:00, 5106.56it/s]    
Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  36%|███▌      | 254/710 [00:00<00:00, 5117.24it/s]
Saving model.layers.0.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  36%|███▌      | 255/710 [00:00<00:00, 4878.08it/s]    
Saving model.layers.0.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  36%|███▌      | 256/710 [00:00<00:00, 4701.64it/s]    
Saving model.layers.0.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  36%|███▌      | 257/710 [00:00<00:00, 4683.62it/s]
Saving model.layers.0.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  36%|███▋      | 258/710 [00:00<00:00, 4430.35it/s]
Saving model.layers.0.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  36%|███▋      | 259/710 [00:00<00:00, 4431.03it/s]     
Saving model.layers.0.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  37%|███▋      | 260/710 [00:00<00:00, 4420.50it/s]
Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  37%|███▋      | 261/710 [00:00<00:00, 4373.83it/s]     
Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  37%|███▋      | 262/710 [00:00<00:00, 4095.21it/s]    
Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  37%|███▋      | 263/710 [00:00<00:00, 4061.45it/s]
Saving model.layers.0.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  37%|███▋      | 264/710 [00:00<00:00, 3789.18it/s]      
Saving model.layers.0.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  37%|███▋      | 265/710 [00:00<00:00, 3753.62it/s]    
Saving model.layers.0.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  37%|███▋      | 266/710 [00:00<00:00, 3759.87it/s]
Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  38%|███▊      | 267/710 [00:00<00:00, 3628.40it/s]
Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  38%|███▊      | 268/710 [00:00<00:00, 3542.84it/s]    
Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  38%|███▊      | 269/710 [00:00<00:00, 3549.09it/s]
Saving model.layers.1.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  38%|███▊      | 270/710 [00:00<00:00, 3453.72it/s]    
Saving model.layers.1.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  38%|███▊      | 271/710 [00:00<00:00, 3438.66it/s]    
Saving model.layers.1.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  38%|███▊      | 272/710 [00:00<00:00, 3448.50it/s]
Saving model.layers.1.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  38%|███▊      | 273/710 [00:00<00:00, 3279.47it/s]
Saving model.layers.1.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  39%|███▊      | 274/710 [00:00<00:00, 3279.16it/s]     
Saving model.layers.1.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  39%|███▊      | 275/710 [00:00<00:00, 3287.58it/s]
Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  39%|███▉      | 276/710 [00:00<00:00, 3255.83it/s]     
Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  39%|███▉      | 277/710 [00:00<00:00, 3229.37it/s]    
Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  39%|███▉      | 278/710 [00:00<00:00, 3227.25it/s]
Saving model.layers.1.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  39%|███▉      | 279/710 [00:00<00:00, 3001.17it/s]      
Saving model.layers.1.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  39%|███▉      | 280/710 [00:00<00:00, 2987.94it/s]    
Saving model.layers.1.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  40%|███▉      | 281/710 [00:00<00:00, 2991.11it/s]
Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  40%|███▉      | 282/710 [00:00<00:00, 2907.90it/s]
Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  40%|███▉      | 283/710 [00:00<00:00, 2879.46it/s]    
Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  40%|████      | 284/710 [00:00<00:00, 2881.08it/s]
Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  40%|████      | 285/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.2.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  40%|████      | 285/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.2.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  40%|████      | 286/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.2.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  40%|████      | 287/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.2.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  41%|████      | 288/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.2.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  41%|████      | 289/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.2.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  41%|████      | 290/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  41%|████      | 291/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  41%|████      | 292/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  41%|████▏     | 293/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.2.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  41%|████▏     | 294/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.2.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  42%|████▏     | 295/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.2.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  42%|████▏     | 296/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.3.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  42%|████▏     | 297/710 [00:00<00:00, 2821.50it/s]  
Saving model.layers.3.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  42%|████▏     | 298/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.3.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  42%|████▏     | 299/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.3.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  42%|████▏     | 300/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.3.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  42%|████▏     | 301/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.3.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  43%|████▎     | 302/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  43%|████▎     | 303/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  43%|████▎     | 304/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  43%|████▎     | 305/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.3.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  43%|████▎     | 306/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.3.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  43%|████▎     | 307/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.3.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  43%|████▎     | 308/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  44%|████▎     | 309/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  44%|████▎     | 310/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  44%|████▍     | 311/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.4.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  44%|████▍     | 312/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.4.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  44%|████▍     | 313/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.4.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  44%|████▍     | 314/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.4.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  44%|████▍     | 315/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.4.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  45%|████▍     | 316/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.4.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  45%|████▍     | 317/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  45%|████▍     | 318/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  45%|████▍     | 319/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  45%|████▌     | 320/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.4.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  45%|████▌     | 321/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.4.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  45%|████▌     | 322/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.4.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  45%|████▌     | 323/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  46%|████▌     | 324/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  46%|████▌     | 325/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  46%|████▌     | 326/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.5.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  46%|████▌     | 327/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.5.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  46%|████▌     | 328/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.5.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  46%|████▋     | 329/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.5.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  46%|████▋     | 330/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.5.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  47%|████▋     | 331/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.5.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  47%|████▋     | 332/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  47%|████▋     | 333/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  47%|████▋     | 334/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  47%|████▋     | 335/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.5.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  47%|████▋     | 336/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.5.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  47%|████▋     | 337/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.5.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  48%|████▊     | 338/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  48%|████▊     | 339/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  48%|████▊     | 340/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  48%|████▊     | 341/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.6.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  48%|████▊     | 342/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.6.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  48%|████▊     | 343/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.6.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  48%|████▊     | 344/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.6.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  49%|████▊     | 345/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.6.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  49%|████▊     | 346/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.6.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  49%|████▉     | 347/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  49%|████▉     | 348/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  49%|████▉     | 349/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  49%|████▉     | 350/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.6.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  49%|████▉     | 351/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.6.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  50%|████▉     | 352/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.6.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  50%|████▉     | 353/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.7.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  50%|████▉     | 354/710 [00:00<00:00, 2821.50it/s]  
Saving model.layers.7.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  50%|█████     | 355/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.7.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  50%|█████     | 356/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.7.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  50%|█████     | 357/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.7.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  50%|█████     | 358/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.7.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  51%|█████     | 359/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  51%|█████     | 360/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  51%|█████     | 361/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  51%|█████     | 362/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.7.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  51%|█████     | 363/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.7.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  51%|█████▏    | 364/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.7.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  51%|█████▏    | 365/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  52%|█████▏    | 366/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  52%|█████▏    | 367/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  52%|█████▏    | 368/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.8.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  52%|█████▏    | 369/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.8.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  52%|█████▏    | 370/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.8.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  52%|█████▏    | 371/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.8.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  52%|█████▏    | 372/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.8.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  53%|█████▎    | 373/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.8.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  53%|█████▎    | 374/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  53%|█████▎    | 375/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  53%|█████▎    | 376/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  53%|█████▎    | 377/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.8.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  53%|█████▎    | 378/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.8.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  53%|█████▎    | 379/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.8.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  54%|█████▎    | 380/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  54%|█████▎    | 381/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  54%|█████▍    | 382/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  54%|█████▍    | 383/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.9.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  54%|█████▍    | 384/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.9.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  54%|█████▍    | 385/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.9.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  54%|█████▍    | 386/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.9.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  55%|█████▍    | 387/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.9.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  55%|█████▍    | 388/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.9.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  55%|█████▍    | 389/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  55%|█████▍    | 390/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  55%|█████▌    | 391/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  55%|█████▌    | 392/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.9.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  55%|█████▌    | 393/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.9.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  55%|█████▌    | 394/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.9.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  56%|█████▌    | 395/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  56%|█████▌    | 396/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  56%|█████▌    | 397/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  56%|█████▌    | 398/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.10.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  56%|█████▌    | 399/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.10.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  56%|█████▋    | 400/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.10.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  56%|█████▋    | 401/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.10.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  57%|█████▋    | 402/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.10.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  57%|█████▋    | 403/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.10.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  57%|█████▋    | 404/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  57%|█████▋    | 405/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  57%|█████▋    | 406/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  57%|█████▋    | 407/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.10.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  57%|█████▋    | 408/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.10.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  58%|█████▊    | 409/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.10.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  58%|█████▊    | 410/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.11.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  58%|█████▊    | 411/710 [00:00<00:00, 2821.50it/s]  
Saving model.layers.11.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  58%|█████▊    | 412/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.11.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  58%|█████▊    | 413/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.11.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  58%|█████▊    | 414/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.11.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  58%|█████▊    | 415/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.11.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  59%|█████▊    | 416/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  59%|█████▊    | 417/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  59%|█████▉    | 418/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  59%|█████▉    | 419/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.11.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  59%|█████▉    | 420/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.11.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  59%|█████▉    | 421/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.11.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  59%|█████▉    | 422/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  60%|█████▉    | 423/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  60%|█████▉    | 424/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  60%|█████▉    | 425/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.12.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  60%|██████    | 426/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.12.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  60%|██████    | 427/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.12.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  60%|██████    | 428/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.12.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  60%|██████    | 429/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.12.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  61%|██████    | 430/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.12.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  61%|██████    | 431/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  61%|██████    | 432/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  61%|██████    | 433/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  61%|██████    | 434/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.12.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  61%|██████▏   | 435/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.12.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  61%|██████▏   | 436/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.12.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  62%|██████▏   | 437/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  62%|██████▏   | 438/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  62%|██████▏   | 439/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  62%|██████▏   | 440/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.13.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  62%|██████▏   | 441/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.13.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  62%|██████▏   | 442/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.13.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  62%|██████▏   | 443/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.13.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  63%|██████▎   | 444/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.13.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  63%|██████▎   | 445/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.13.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  63%|██████▎   | 446/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  63%|██████▎   | 447/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  63%|██████▎   | 448/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  63%|██████▎   | 449/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.13.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  63%|██████▎   | 450/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.13.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  64%|██████▎   | 451/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.13.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  64%|██████▎   | 452/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  64%|██████▍   | 453/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  64%|██████▍   | 454/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  64%|██████▍   | 455/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.14.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  64%|██████▍   | 456/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.14.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  64%|██████▍   | 457/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.14.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  65%|██████▍   | 458/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.14.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  65%|██████▍   | 459/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.14.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  65%|██████▍   | 460/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.14.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  65%|██████▍   | 461/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  65%|██████▌   | 462/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  65%|██████▌   | 463/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  65%|██████▌   | 464/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.14.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  65%|██████▌   | 465/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.14.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  66%|██████▌   | 466/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.14.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  66%|██████▌   | 467/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.15.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  66%|██████▌   | 468/710 [00:00<00:00, 2821.50it/s]  
Saving model.layers.15.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  66%|██████▌   | 469/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.15.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  66%|██████▌   | 470/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.15.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  66%|██████▋   | 471/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.15.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  66%|██████▋   | 472/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.15.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  67%|██████▋   | 473/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  67%|██████▋   | 474/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  67%|██████▋   | 475/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  67%|██████▋   | 476/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.15.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  67%|██████▋   | 477/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.15.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  67%|██████▋   | 478/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.15.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  67%|██████▋   | 479/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  68%|██████▊   | 480/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  68%|██████▊   | 481/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  68%|██████▊   | 482/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.16.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  68%|██████▊   | 483/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.16.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  68%|██████▊   | 484/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.16.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  68%|██████▊   | 485/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.16.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  68%|██████▊   | 486/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.16.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  69%|██████▊   | 487/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.16.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  69%|██████▊   | 488/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  69%|██████▉   | 489/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  69%|██████▉   | 490/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  69%|██████▉   | 491/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.16.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  69%|██████▉   | 492/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.16.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  69%|██████▉   | 493/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.16.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  70%|██████▉   | 494/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  70%|██████▉   | 495/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  70%|██████▉   | 496/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  70%|███████   | 497/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.17.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  70%|███████   | 498/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.17.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  70%|███████   | 499/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.17.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  70%|███████   | 500/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.17.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  71%|███████   | 501/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.17.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  71%|███████   | 502/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.17.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  71%|███████   | 503/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  71%|███████   | 504/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  71%|███████   | 505/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  71%|███████▏  | 506/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.17.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  71%|███████▏  | 507/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.17.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  72%|███████▏  | 508/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.17.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  72%|███████▏  | 509/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  72%|███████▏  | 510/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  72%|███████▏  | 511/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  72%|███████▏  | 512/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.18.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  72%|███████▏  | 513/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.18.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  72%|███████▏  | 514/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.18.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  73%|███████▎  | 515/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.18.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  73%|███████▎  | 516/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.18.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  73%|███████▎  | 517/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.18.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  73%|███████▎  | 518/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  73%|███████▎  | 519/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  73%|███████▎  | 520/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  73%|███████▎  | 521/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.18.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  74%|███████▎  | 522/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.18.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  74%|███████▎  | 523/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.18.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  74%|███████▍  | 524/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.19.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  74%|███████▍  | 525/710 [00:00<00:00, 2821.50it/s]  
Saving model.layers.19.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  74%|███████▍  | 526/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.19.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  74%|███████▍  | 527/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.19.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  74%|███████▍  | 528/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.19.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  75%|███████▍  | 529/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.19.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  75%|███████▍  | 530/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  75%|███████▍  | 531/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  75%|███████▍  | 532/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  75%|███████▌  | 533/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.19.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  75%|███████▌  | 534/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.19.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  75%|███████▌  | 535/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.19.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  75%|███████▌  | 536/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  76%|███████▌  | 537/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  76%|███████▌  | 538/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  76%|███████▌  | 539/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.20.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  76%|███████▌  | 540/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.20.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  76%|███████▌  | 541/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.20.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  76%|███████▋  | 542/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.20.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  76%|███████▋  | 543/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.20.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  77%|███████▋  | 544/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.20.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  77%|███████▋  | 545/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  77%|███████▋  | 546/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  77%|███████▋  | 547/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  77%|███████▋  | 548/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.20.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  77%|███████▋  | 549/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.20.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  77%|███████▋  | 550/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.20.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  78%|███████▊  | 551/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  78%|███████▊  | 552/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  78%|███████▊  | 553/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  78%|███████▊  | 554/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.21.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  78%|███████▊  | 555/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.21.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  78%|███████▊  | 556/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.21.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  78%|███████▊  | 557/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.21.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  79%|███████▊  | 558/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.21.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  79%|███████▊  | 559/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.21.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  79%|███████▉  | 560/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  79%|███████▉  | 561/710 [00:00<00:00, 2821.50it/s]     
Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  79%|███████▉  | 562/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  79%|███████▉  | 563/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.21.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  79%|███████▉  | 564/710 [00:00<00:00, 2821.50it/s]      
Saving model.layers.21.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  80%|███████▉  | 565/710 [00:00<00:00, 2821.50it/s]    
Saving model.layers.21.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  80%|███████▉  | 566/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  80%|███████▉  | 567/710 [00:00<00:00, 2821.50it/s]
Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  80%|████████  | 568/710 [00:00<00:00, 497.89it/s] 
Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  80%|████████  | 568/710 [00:00<00:00, 497.89it/s]    
Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  80%|████████  | 569/710 [00:00<00:00, 497.89it/s]
Saving model.layers.22.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  80%|████████  | 570/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.22.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  80%|████████  | 571/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.22.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  81%|████████  | 572/710 [00:01<00:00, 497.89it/s]
Saving model.layers.22.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  81%|████████  | 573/710 [00:01<00:00, 497.89it/s]
Saving model.layers.22.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  81%|████████  | 574/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.22.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  81%|████████  | 575/710 [00:01<00:00, 497.89it/s]
Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  81%|████████  | 576/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  81%|████████▏ | 577/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  81%|████████▏ | 578/710 [00:01<00:00, 497.89it/s]
Saving model.layers.22.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  82%|████████▏ | 579/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.22.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  82%|████████▏ | 580/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.22.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  82%|████████▏ | 581/710 [00:01<00:00, 497.89it/s]
Saving model.layers.23.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  82%|████████▏ | 582/710 [00:01<00:00, 497.89it/s]  
Saving model.layers.23.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  82%|████████▏ | 583/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.23.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  82%|████████▏ | 584/710 [00:01<00:00, 497.89it/s]
Saving model.layers.23.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  82%|████████▏ | 585/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.23.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  83%|████████▎ | 586/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.23.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  83%|████████▎ | 587/710 [00:01<00:00, 497.89it/s]
Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  83%|████████▎ | 588/710 [00:01<00:00, 497.89it/s]
Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  83%|████████▎ | 589/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  83%|████████▎ | 590/710 [00:01<00:00, 497.89it/s]
Saving model.layers.23.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  83%|████████▎ | 591/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.23.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  83%|████████▎ | 592/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.23.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  84%|████████▎ | 593/710 [00:01<00:00, 497.89it/s]
Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  84%|████████▎ | 594/710 [00:01<00:00, 497.89it/s]
Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  84%|████████▍ | 595/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  84%|████████▍ | 596/710 [00:01<00:00, 497.89it/s]
Saving model.layers.24.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  84%|████████▍ | 597/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.24.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  84%|████████▍ | 598/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.24.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  84%|████████▍ | 599/710 [00:01<00:00, 497.89it/s]
Saving model.layers.24.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  85%|████████▍ | 600/710 [00:01<00:00, 497.89it/s]
Saving model.layers.24.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  85%|████████▍ | 601/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.24.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  85%|████████▍ | 602/710 [00:01<00:00, 497.89it/s]
Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  85%|████████▍ | 603/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  85%|████████▌ | 604/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  85%|████████▌ | 605/710 [00:01<00:00, 497.89it/s]
Saving model.layers.24.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  85%|████████▌ | 606/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.24.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  85%|████████▌ | 607/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.24.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  86%|████████▌ | 608/710 [00:01<00:00, 497.89it/s]
Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  86%|████████▌ | 609/710 [00:01<00:00, 497.89it/s]
Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  86%|████████▌ | 610/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  86%|████████▌ | 611/710 [00:01<00:00, 497.89it/s]
Saving model.layers.25.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  86%|████████▌ | 612/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.25.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  86%|████████▋ | 613/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.25.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  86%|████████▋ | 614/710 [00:01<00:00, 497.89it/s]
Saving model.layers.25.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  87%|████████▋ | 615/710 [00:01<00:00, 497.89it/s]
Saving model.layers.25.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  87%|████████▋ | 616/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.25.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  87%|████████▋ | 617/710 [00:01<00:00, 497.89it/s]
Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  87%|████████▋ | 618/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  87%|████████▋ | 619/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  87%|████████▋ | 620/710 [00:01<00:00, 497.89it/s]
Saving model.layers.25.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  87%|████████▋ | 621/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.25.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  88%|████████▊ | 622/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.25.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  88%|████████▊ | 623/710 [00:01<00:00, 497.89it/s]
Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  88%|████████▊ | 624/710 [00:01<00:00, 497.89it/s]
Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  88%|████████▊ | 625/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  88%|████████▊ | 626/710 [00:01<00:00, 497.89it/s]
Saving model.layers.26.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  88%|████████▊ | 627/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.26.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  88%|████████▊ | 628/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.26.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  89%|████████▊ | 629/710 [00:01<00:00, 497.89it/s]
Saving model.layers.26.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  89%|████████▊ | 630/710 [00:01<00:00, 497.89it/s]
Saving model.layers.26.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  89%|████████▉ | 631/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.26.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  89%|████████▉ | 632/710 [00:01<00:00, 497.89it/s]
Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  89%|████████▉ | 633/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  89%|████████▉ | 634/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  89%|████████▉ | 635/710 [00:01<00:00, 497.89it/s]
Saving model.layers.26.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  90%|████████▉ | 636/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.26.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  90%|████████▉ | 637/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.26.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  90%|████████▉ | 638/710 [00:01<00:00, 497.89it/s]
Saving model.layers.27.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  90%|█████████ | 639/710 [00:01<00:00, 497.89it/s]  
Saving model.layers.27.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  90%|█████████ | 640/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.27.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  90%|█████████ | 641/710 [00:01<00:00, 497.89it/s]
Saving model.layers.27.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  90%|█████████ | 642/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.27.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  91%|█████████ | 643/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.27.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  91%|█████████ | 644/710 [00:01<00:00, 497.89it/s]
Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  91%|█████████ | 645/710 [00:01<00:00, 497.89it/s]
Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  91%|█████████ | 646/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  91%|█████████ | 647/710 [00:01<00:00, 497.89it/s]
Saving model.layers.27.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  91%|█████████▏| 648/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.27.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  91%|█████████▏| 649/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.27.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  92%|█████████▏| 650/710 [00:01<00:00, 497.89it/s]
Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  92%|█████████▏| 651/710 [00:01<00:00, 497.89it/s]
Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  92%|█████████▏| 652/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  92%|█████████▏| 653/710 [00:01<00:00, 497.89it/s]
Saving model.layers.28.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  92%|█████████▏| 654/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.28.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  92%|█████████▏| 655/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.28.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  92%|█████████▏| 656/710 [00:01<00:00, 497.89it/s]
Saving model.layers.28.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  93%|█████████▎| 657/710 [00:01<00:00, 497.89it/s]
Saving model.layers.28.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  93%|█████████▎| 658/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.28.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  93%|█████████▎| 659/710 [00:01<00:00, 497.89it/s]
Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  93%|█████████▎| 660/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  93%|█████████▎| 661/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  93%|█████████▎| 662/710 [00:01<00:00, 497.89it/s]
Saving model.layers.28.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  93%|█████████▎| 663/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.28.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  94%|█████████▎| 664/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.28.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  94%|█████████▎| 665/710 [00:01<00:00, 497.89it/s]
Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  94%|█████████▍| 666/710 [00:01<00:00, 497.89it/s]
Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  94%|█████████▍| 667/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  94%|█████████▍| 668/710 [00:01<00:00, 497.89it/s]
Saving model.layers.29.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  94%|█████████▍| 669/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.29.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  94%|█████████▍| 670/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.29.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  95%|█████████▍| 671/710 [00:01<00:00, 497.89it/s]
Saving model.layers.29.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  95%|█████████▍| 672/710 [00:01<00:00, 497.89it/s]
Saving model.layers.29.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  95%|█████████▍| 673/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.29.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  95%|█████████▍| 674/710 [00:01<00:00, 497.89it/s]
Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  95%|█████████▌| 675/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  95%|█████████▌| 676/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  95%|█████████▌| 677/710 [00:01<00:00, 497.89it/s]
Saving model.layers.29.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  95%|█████████▌| 678/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.29.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  96%|█████████▌| 679/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.29.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  96%|█████████▌| 680/710 [00:01<00:00, 497.89it/s]
Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_scale (f16, [8192,80]):  96%|█████████▌| 681/710 [00:01<00:00, 497.89it/s]
Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_zp (u8, [8192,40]):  96%|█████████▌| 682/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_Q4G32 (u8, [8192,80,16]):  96%|█████████▌| 683/710 [00:01<00:00, 497.89it/s]
Saving model.layers.30.linear_attn.z_proj.MatMul.weight_scale (f16, [4096,80]):  96%|█████████▋| 684/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.30.linear_attn.z_proj.MatMul.weight_zp (u8, [4096,40]):  96%|█████████▋| 685/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.30.linear_attn.z_proj.MatMul.weight_Q4G32 (u8, [4096,80,16]):  97%|█████████▋| 686/710 [00:01<00:00, 497.89it/s]
Saving model.layers.30.linear_attn.out_proj.MatMul.weight_scale (f16, [2560,128]):  97%|█████████▋| 687/710 [00:01<00:00, 497.89it/s]
Saving model.layers.30.linear_attn.out_proj.MatMul.weight_zp (u8, [2560,64]):  97%|█████████▋| 688/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.30.linear_attn.out_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  97%|█████████▋| 689/710 [00:01<00:00, 497.89it/s]
Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  97%|█████████▋| 690/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  97%|█████████▋| 691/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  97%|█████████▋| 692/710 [00:01<00:00, 497.89it/s]
Saving model.layers.30.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  98%|█████████▊| 693/710 [00:01<00:00, 497.89it/s]      
Saving model.layers.30.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  98%|█████████▊| 694/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.30.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  98%|█████████▊| 695/710 [00:01<00:00, 497.89it/s]
Saving model.layers.31.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  98%|█████████▊| 696/710 [00:01<00:00, 497.89it/s]  
Saving model.layers.31.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  98%|█████████▊| 697/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.31.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  98%|█████████▊| 698/710 [00:01<00:00, 497.89it/s]
Saving model.layers.31.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  98%|█████████▊| 699/710 [00:01<00:00, 497.89it/s]    
Saving model.layers.31.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  99%|█████████▊| 700/710 [00:01<00:00, 497.89it/s]     
Saving model.layers.31.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  99%|█████████▊| 701/710 [00:01<00:00, 497.89it/s]
Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  99%|█████████▉| 702/710 [00:01<00:00, 497.89it/s]
Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  99%|█████████▉| 703/710 [00:01<00:00, 393.44it/s]
Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  99%|█████████▉| 703/710 [00:01<00:00, 393.44it/s]    
Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  99%|█████████▉| 704/710 [00:01<00:00, 393.44it/s]
Saving model.layers.31.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  99%|█████████▉| 705/710 [00:01<00:00, 393.44it/s]      
Saving model.layers.31.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  99%|█████████▉| 706/710 [00:01<00:00, 393.44it/s]    
Saving model.layers.31.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]): 100%|█████████▉| 707/710 [00:01<00:00, 393.44it/s]
Saving lm_head.MatMul.weight_scale (f16, [248320,80]): 100%|█████████▉| 708/710 [00:01<00:00, 393.44it/s]                       
Saving lm_head.MatMul.weight_zp (u8, [248320,80]): 100%|█████████▉| 709/710 [00:01<00:00, 393.44it/s]    
Saving lm_head.MatMul.weight_Q8G32 (u8, [248320,80,32]): 100%|██████████| 710/710 [00:01<00:00, 393.44it/s]
Saving lm_head.MatMul.weight_Q8G32 (u8, [248320,80,32]): 100%|██████████| 710/710 [00:01<00:00, 368.23it/s]
2026-11-05 14:19:58,118 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize model with k_quant algorithm...
2026-11-05 14:19:58,119 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_Q8G32' of node '/model/mtp/embed_tokens/Reshape' (::Reshape:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-05 14:19:58,119 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_scale' of node '/model/mtp/embed_tokens/GatherBlockQuantized' (com.microsoft::GatherBlockQuantized:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-05 14:19:58,119 onnx_ir.serde [WARNING] - Input 'lm_head.MatMul.weight_zp' of node '/model/mtp/embed_tokens/GatherBlockQuantized' (com.microsoft::GatherBlockQuantized:) cannot be found in any scope. The model is invalid but we will still create a new input for the node (current depth: 1)
2026-11-05 14:19:58,133 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:20:00,299 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:20:04,599 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:20:06,348 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:20:13,798 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:20:17,998 neural_compressor [INFO] - Now we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.
2026-11-05 14:22:31,761 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of model with k_quant algorithm.
Saving ONNX model in model-cuda-int4-slim

Progress: [###                 ] 16.67%
Progress: [######              ] 33.33%
Progress: [##########          ] 50.00%
Progress: [#############       ] 66.67%
Progress: [################    ] 83.33%
Progress: [####################] 100.00%
0it [00:00, ?it/s]
Saving model.mtp.pre_fc_norm_embedding.weight (f16, [2560]):   4%|▎         | 1/27 [00:00<00:00, 687.82it/s]
Saving model.mtp.pre_fc_norm_hidden.weight (f16, [2560]):   7%|▋         | 2/27 [00:00<00:00, 1307.86it/s]  
Saving model.layers.0.input_layernorm.weight (f16, [2560]):  11%|█         | 3/27 [00:00<00:00, 1902.47it/s]
Saving model.layers.0.attn.q_norm.layernorm.weight (f16, [256]):  15%|█▍        | 4/27 [00:00<00:00, 2482.20it/s]
Saving model.layers.0.attn.k_norm.layernorm.weight (f16, [256]):  19%|█▊        | 5/27 [00:00<00:00, 3053.51it/s]
Saving cos_cache (f32, [262144,32]):  22%|██▏       | 6/27 [00:00<00:00, 3603.35it/s]                            
Saving sin_cache (f32, [262144,32]):  26%|██▌       | 7/27 [00:00<00:00, 832.91it/s] 
Saving model.layers.0.post_attention_layernorm.weight (f16, [2560]):  30%|██▉       | 8/27 [00:00<00:00, 582.41it/s]
Saving model.layers.1.final_norm_layernorm.weight (f16, [2560]):  33%|███▎      | 9/27 [00:00<00:00, 630.27it/s]    
Saving model.mtp.fc.MatMul.weight_scale (f16, [2560,160]):  37%|███▋      | 10/27 [00:00<00:00, 696.52it/s]     
Saving model.mtp.fc.MatMul.weight_zp (u8, [2560,80]):  41%|████      | 11/27 [00:00<00:00, 755.80it/s]     
Saving model.mtp.fc.MatMul.weight_Q4G32 (u8, [2560,160,16]):  44%|████▍     | 12/27 [00:00<00:00, 819.51it/s]
Saving model.layers.0.attn.qkv_proj.MatMul.weight_scale (f16, [10240,80]):  48%|████▊     | 13/27 [00:00<00:00, 815.27it/s]
Saving model.layers.0.attn.qkv_proj.MatMul.weight_zp (u8, [10240,40]):  52%|█████▏    | 14/27 [00:00<00:00, 858.61it/s]    
Saving model.layers.0.attn.qkv_proj.MatMul.weight_Q4G32 (u8, [10240,80,16]):  56%|█████▌    | 15/27 [00:00<00:00, 913.46it/s]
Saving model.layers.0.attn.o_proj.MatMul.weight_scale (f16, [2560,128]):  59%|█████▉    | 16/27 [00:00<00:00, 863.96it/s]    
Saving model.layers.0.attn.o_proj.MatMul.weight_zp (u8, [2560,64]):  63%|██████▎   | 17/27 [00:00<00:00, 909.46it/s]     
Saving model.layers.0.attn.o_proj.MatMul.weight_Q4G32 (u8, [2560,128,16]):  67%|██████▋   | 18/27 [00:00<00:00, 959.49it/s]
Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scale (f16, [18432,80]):  70%|███████   | 19/27 [00:00<00:00, 966.31it/s]
Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_zp (u8, [18432,40]):  74%|███████▍  | 20/27 [00:00<00:00, 994.91it/s]    
Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_Q4G32 (u8, [18432,80,16]):  78%|███████▊  | 21/27 [00:00<00:00, 1034.58it/s]
Saving model.layers.0.mlp.down_proj.MatMul.weight_scale (f16, [2560,288]):  81%|████████▏ | 22/27 [00:00<00:00, 886.33it/s]       
Saving model.layers.0.mlp.down_proj.MatMul.weight_zp (u8, [2560,144]):  85%|████████▌ | 23/27 [00:00<00:00, 913.48it/s]    
Saving model.layers.0.mlp.down_proj.MatMul.weight_Q4G32 (u8, [2560,288,16]):  89%|████████▉ | 24/27 [00:00<00:00, 949.32it/s]
Saving lm_head.MatMul.weight_scale (f16, [248320,80]):  93%|█████████▎| 25/27 [00:00<00:00, 919.36it/s]                      
Saving lm_head.MatMul.weight_zp (u8, [248320,80]):  96%|█████████▋| 26/27 [00:00<00:00, 727.40it/s]    
Saving lm_head.MatMul.weight_Q8G32 (u8, [248320,80,32]): 100%|██████████| 27/27 [00:00<00:00, 676.38it/s]
Saving lm_head.MatMul.weight_Q8G32 (u8, [248320,80,32]): 100%|██████████| 27/27 [00:00<00:00, 71.33it/s] 
Configured 695 MB of runtime initializer sharing; model-cuda-int4-slim/mtp.onnx.data remains self-contained.
Warning: could not read generation_config.json (/home/anon/LLM/KnowLine-4B-Gen5.5 does not appear to have a file named generation_config.json. Checkout 'https://huggingface.co//home/anon/LLM/KnowLine-4B-Gen5.5/tree/main' for available files.). Falling back to config.json.
Adding the tokenizer's end-of-turn token <|im_end|> (id 248046) to eos_token_id from the model configuration: 248044.
Saving GenAI config in model-cuda-int4-slim
Added 'mtp' section to genai_config.json
Saving processing files in model-cuda-int4-slim for GenAI
EXIT:0