Valid precision + execution provider combinations are: FP32 CPU, FP32 CUDA, FP16 CUDA, FP16 DML, FP16 TRT-RTX, BF16 CUDA, BF16 TRT-RTX, INT8 CPU, INT8 CUDA, INT8 WebGPU, INT4 CPU, INT4 CUDA, INT4 DML, INT4 WebGPU Extra options: {'accuracy_level': '4', 'block_size': '32'} GroupQueryAttention (GQA) is used in this model. GroupQueryAttention (GQA) is used in this model. Loading weights: 0%| | 0/426 [00:00 mtp.onnx Saving ONNX model in /home/anon/knowline/onnx/model-cpu 0it [00:00, ?it/s] Saving model.layers.0.input_layernorm.weight (f32, [2560]): 0%| | 1/557 [00:00<00:04, 132.46it/s] Saving model.layers.0.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 0%| | 2/557 [00:00<00:02, 261.56it/s] Saving model.layers.0.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 1%| | 3/557 [00:00<00:01, 383.04it/s] Saving model.layers.0.linear_attn.conv1d.weight (f32, [8192,1,4]): 1%| | 4/557 [00:00<00:01, 504.65it/s] Saving model.layers.0.linear_attn.conv1d.bias (f32, [8192]): 1%| | 5/557 [00:00<00:00, 625.77it/s] Saving model.layers.0.linear_attn.dt_bias (f32, [32]): 1%| | 6/557 [00:00<00:00, 747.03it/s] Saving model.layers.0.linear_attn.neg_exp_A (f32, [32]): 1%|▏ | 7/557 [00:00<00:00, 869.00it/s] Saving model.layers.0.linear_attn.norm.weight (f32, [128]): 1%|▏ | 8/557 [00:00<00:00, 989.69it/s] Saving model.layers.0.post_attention_layernorm.weight (f32, [2560]): 2%|▏ | 9/557 [00:00<00:00, 1109.93it/s] Saving model.layers.1.input_layernorm.weight (f32, [2560]): 2%|▏ | 10/557 [00:00<00:00, 1228.06it/s] Saving model.layers.1.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 2%|▏ | 11/557 [00:00<00:00, 1345.58it/s] Saving model.layers.1.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 2%|▏ | 12/557 [00:00<00:00, 1453.37it/s] Saving model.layers.1.linear_attn.conv1d.weight (f32, [8192,1,4]): 2%|▏ | 13/557 [00:00<00:00, 1559.04it/s] Saving model.layers.1.linear_attn.conv1d.bias (f32, [8192]): 3%|▎ | 14/557 [00:00<00:00, 1657.92it/s] Saving model.layers.1.linear_attn.dt_bias (f32, [32]): 3%|▎ | 15/557 [00:00<00:00, 1767.46it/s] Saving model.layers.1.linear_attn.neg_exp_A (f32, [32]): 3%|▎ | 16/557 [00:00<00:00, 1880.27it/s] Saving model.layers.1.linear_attn.norm.weight (f32, [128]): 3%|▎ | 17/557 [00:00<00:00, 1991.93it/s] Saving model.layers.1.post_attention_layernorm.weight (f32, [2560]): 3%|▎ | 18/557 [00:00<00:00, 2103.29it/s] Saving model.layers.2.input_layernorm.weight (f32, [2560]): 3%|▎ | 19/557 [00:00<00:00, 2212.31it/s] Saving model.layers.2.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 4%|▎ | 20/557 [00:00<00:00, 2321.14it/s] Saving model.layers.2.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 4%|▍ | 21/557 [00:00<00:00, 2415.48it/s] Saving model.layers.2.linear_attn.conv1d.weight (f32, [8192,1,4]): 4%|▍ | 22/557 [00:00<00:00, 2505.56it/s] Saving model.layers.2.linear_attn.conv1d.bias (f32, [8192]): 4%|▍ | 23/557 [00:00<00:00, 2603.61it/s] Saving model.layers.2.linear_attn.dt_bias (f32, [32]): 4%|▍ | 24/557 [00:00<00:00, 2705.78it/s] Saving model.layers.2.linear_attn.neg_exp_A (f32, [32]): 4%|▍ | 25/557 [00:00<00:00, 2811.35it/s] Saving model.layers.2.linear_attn.norm.weight (f32, [128]): 5%|▍ | 26/557 [00:00<00:00, 2915.83it/s] Saving model.layers.2.post_attention_layernorm.weight (f32, [2560]): 5%|▍ | 27/557 [00:00<00:00, 3020.22it/s] Saving model.layers.3.input_layernorm.weight (f32, [2560]): 5%|▌ | 28/557 [00:00<00:00, 3120.68it/s] Saving model.layers.3.attn.q_norm.layernorm.weight (f32, [256]): 5%|▌ | 29/557 [00:00<00:00, 3221.18it/s] Saving model.layers.3.attn.k_norm.layernorm.weight (f32, [256]): 5%|▌ | 30/557 [00:00<00:00, 3323.80it/s] Saving cos_cache (f32, [262144,32]): 6%|▌ | 31/557 [00:00<00:00, 3425.46it/s] Saving sin_cache (f32, [262144,32]): 6%|▌ | 32/557 [00:00<00:00, 670.12it/s] Saving model.layers.3.post_attention_layernorm.weight (f32, [2560]): 6%|▌ | 33/557 [00:00<00:01, 475.07it/s] Saving model.layers.4.input_layernorm.weight (f32, [2560]): 6%|▌ | 34/557 [00:00<00:01, 488.71it/s] Saving model.layers.4.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 6%|▋ | 35/557 [00:00<00:01, 502.78it/s] Saving model.layers.4.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 6%|▋ | 36/557 [00:00<00:01, 516.44it/s] Saving model.layers.4.linear_attn.conv1d.weight (f32, [8192,1,4]): 7%|▋ | 37/557 [00:00<00:00, 530.16it/s] Saving model.layers.4.linear_attn.conv1d.bias (f32, [8192]): 7%|▋ | 38/557 [00:00<00:00, 544.05it/s] Saving model.layers.4.linear_attn.dt_bias (f32, [32]): 7%|▋ | 39/557 [00:00<00:00, 558.07it/s] Saving model.layers.4.linear_attn.neg_exp_A (f32, [32]): 7%|▋ | 40/557 [00:00<00:00, 572.20it/s] Saving model.layers.4.linear_attn.norm.weight (f32, [128]): 7%|▋ | 41/557 [00:00<00:00, 586.30it/s] Saving model.layers.4.post_attention_layernorm.weight (f32, [2560]): 8%|▊ | 42/557 [00:00<00:00, 600.41it/s] Saving model.layers.5.input_layernorm.weight (f32, [2560]): 8%|▊ | 43/557 [00:00<00:00, 614.43it/s] Saving model.layers.5.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 8%|▊ | 44/557 [00:00<00:00, 628.44it/s] Saving model.layers.5.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 8%|▊ | 45/557 [00:00<00:00, 642.04it/s] Saving model.layers.5.linear_attn.conv1d.weight (f32, [8192,1,4]): 8%|▊ | 46/557 [00:00<00:00, 655.52it/s] Saving model.layers.5.linear_attn.conv1d.bias (f32, [8192]): 8%|▊ | 47/557 [00:00<00:00, 669.16it/s] Saving model.layers.5.linear_attn.dt_bias (f32, [32]): 9%|▊ | 48/557 [00:00<00:00, 683.01it/s] Saving model.layers.5.linear_attn.neg_exp_A (f32, [32]): 9%|▉ | 49/557 [00:00<00:00, 697.02it/s] Saving model.layers.5.linear_attn.norm.weight (f32, [128]): 9%|▉ | 50/557 [00:00<00:00, 711.01it/s] Saving model.layers.5.post_attention_layernorm.weight (f32, [2560]): 9%|▉ | 51/557 [00:00<00:00, 725.01it/s] Saving model.layers.6.input_layernorm.weight (f32, [2560]): 9%|▉ | 52/557 [00:00<00:00, 738.84it/s] Saving model.layers.6.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 10%|▉ | 53/557 [00:00<00:00, 752.74it/s] Saving model.layers.6.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 10%|▉ | 54/557 [00:00<00:00, 766.19it/s] Saving model.layers.6.linear_attn.conv1d.weight (f32, [8192,1,4]): 10%|▉ | 55/557 [00:00<00:00, 779.53it/s] Saving model.layers.6.linear_attn.conv1d.bias (f32, [8192]): 10%|█ | 56/557 [00:00<00:00, 793.07it/s] Saving model.layers.6.linear_attn.dt_bias (f32, [32]): 10%|█ | 57/557 [00:00<00:00, 806.80it/s] Saving model.layers.6.linear_attn.neg_exp_A (f32, [32]): 10%|█ | 58/557 [00:00<00:00, 820.70it/s] Saving model.layers.6.linear_attn.norm.weight (f32, [128]): 11%|█ | 59/557 [00:00<00:00, 834.57it/s] Saving model.layers.6.post_attention_layernorm.weight (f32, [2560]): 11%|█ | 60/557 [00:00<00:00, 848.45it/s] Saving model.layers.7.input_layernorm.weight (f32, [2560]): 11%|█ | 61/557 [00:00<00:00, 862.23it/s] Saving model.layers.7.attn.q_norm.layernorm.weight (f32, [256]): 11%|█ | 62/557 [00:00<00:00, 876.00it/s] Saving model.layers.7.attn.k_norm.layernorm.weight (f32, [256]): 11%|█▏ | 63/557 [00:00<00:00, 889.88it/s] Saving model.layers.7.post_attention_layernorm.weight (f32, [2560]): 11%|█▏ | 64/557 [00:00<00:00, 903.68it/s] Saving model.layers.8.input_layernorm.weight (f32, [2560]): 12%|█▏ | 65/557 [00:00<00:00, 917.41it/s] Saving model.layers.8.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 12%|█▏ | 66/557 [00:00<00:00, 931.16it/s] Saving model.layers.8.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 12%|█▏ | 67/557 [00:00<00:00, 944.39it/s] Saving model.layers.8.linear_attn.conv1d.weight (f32, [8192,1,4]): 12%|█▏ | 68/557 [00:00<00:00, 957.51it/s] Saving model.layers.8.linear_attn.conv1d.bias (f32, [8192]): 12%|█▏ | 69/557 [00:00<00:00, 970.94it/s] Saving model.layers.8.linear_attn.dt_bias (f32, [32]): 13%|█▎ | 70/557 [00:00<00:00, 984.49it/s] Saving model.layers.8.linear_attn.neg_exp_A (f32, [32]): 13%|█▎ | 71/557 [00:00<00:00, 998.27it/s] Saving model.layers.8.linear_attn.norm.weight (f32, [128]): 13%|█▎ | 72/557 [00:00<00:00, 1012.02it/s] Saving model.layers.8.post_attention_layernorm.weight (f32, [2560]): 13%|█▎ | 73/557 [00:00<00:00, 1025.70it/s] Saving model.layers.9.input_layernorm.weight (f32, [2560]): 13%|█▎ | 74/557 [00:00<00:00, 1039.32it/s] Saving model.layers.9.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 13%|█▎ | 75/557 [00:00<00:00, 1052.97it/s] Saving model.layers.9.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 14%|█▎ | 76/557 [00:00<00:00, 1065.73it/s] Saving model.layers.9.linear_attn.conv1d.weight (f32, [8192,1,4]): 14%|█▍ | 77/557 [00:00<00:00, 1078.41it/s] Saving model.layers.9.linear_attn.conv1d.bias (f32, [8192]): 14%|█▍ | 78/557 [00:00<00:00, 1091.30it/s] Saving model.layers.9.linear_attn.dt_bias (f32, [32]): 14%|█▍ | 79/557 [00:00<00:00, 1104.52it/s] Saving model.layers.9.linear_attn.neg_exp_A (f32, [32]): 14%|█▍ | 80/557 [00:00<00:00, 1118.03it/s] Saving model.layers.9.linear_attn.norm.weight (f32, [128]): 15%|█▍ | 81/557 [00:00<00:00, 1131.50it/s] Saving model.layers.9.post_attention_layernorm.weight (f32, [2560]): 15%|█▍ | 82/557 [00:00<00:00, 1144.97it/s] Saving model.layers.10.input_layernorm.weight (f32, [2560]): 15%|█▍ | 83/557 [00:00<00:00, 1158.29it/s] Saving model.layers.10.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 15%|█▌ | 84/557 [00:00<00:00, 1171.60it/s] Saving model.layers.10.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 15%|█▌ | 85/557 [00:00<00:00, 1184.09it/s] Saving model.layers.10.linear_attn.conv1d.weight (f32, [8192,1,4]): 15%|█▌ | 86/557 [00:00<00:00, 1196.34it/s] Saving model.layers.10.linear_attn.conv1d.bias (f32, [8192]): 16%|█▌ | 87/557 [00:00<00:00, 1209.03it/s] Saving model.layers.10.linear_attn.dt_bias (f32, [32]): 16%|█▌ | 88/557 [00:00<00:00, 1222.12it/s] Saving model.layers.10.linear_attn.neg_exp_A (f32, [32]): 16%|█▌ | 89/557 [00:00<00:00, 1235.49it/s] Saving model.layers.10.linear_attn.norm.weight (f32, [128]): 16%|█▌ | 90/557 [00:00<00:00, 1248.81it/s] Saving model.layers.10.post_attention_layernorm.weight (f32, [2560]): 16%|█▋ | 91/557 [00:00<00:00, 1262.06it/s] Saving model.layers.11.input_layernorm.weight (f32, [2560]): 17%|█▋ | 92/557 [00:00<00:00, 1275.20it/s] Saving model.layers.11.attn.q_norm.layernorm.weight (f32, [256]): 17%|█▋ | 93/557 [00:00<00:00, 1288.27it/s] Saving model.layers.11.attn.k_norm.layernorm.weight (f32, [256]): 17%|█▋ | 94/557 [00:00<00:00, 1301.56it/s] Saving model.layers.11.post_attention_layernorm.weight (f32, [2560]): 17%|█▋ | 95/557 [00:00<00:00, 1314.81it/s] Saving model.layers.12.input_layernorm.weight (f32, [2560]): 17%|█▋ | 96/557 [00:00<00:00, 1327.88it/s] Saving model.layers.12.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 17%|█▋ | 97/557 [00:00<00:00, 1340.93it/s] Saving model.layers.12.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 18%|█▊ | 98/557 [00:00<00:00, 1352.34it/s] Saving model.layers.12.linear_attn.conv1d.weight (f32, [8192,1,4]): 18%|█▊ | 99/557 [00:00<00:00, 1364.33it/s] Saving model.layers.12.linear_attn.conv1d.bias (f32, [8192]): 18%|█▊ | 100/557 [00:00<00:00, 1377.01it/s] Saving model.layers.12.linear_attn.dt_bias (f32, [32]): 18%|█▊ | 101/557 [00:00<00:00, 1390.05it/s] Saving model.layers.12.linear_attn.neg_exp_A (f32, [32]): 18%|█▊ | 102/557 [00:00<00:00, 1403.40it/s] Saving model.layers.12.linear_attn.norm.weight (f32, [128]): 18%|█▊ | 103/557 [00:00<00:00, 1416.71it/s] Saving model.layers.12.post_attention_layernorm.weight (f32, [2560]): 19%|█▊ | 104/557 [00:00<00:00, 1430.03it/s] Saving model.layers.13.input_layernorm.weight (f32, [2560]): 19%|█▉ | 105/557 [00:00<00:00, 1443.19it/s] Saving model.layers.13.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 19%|█▉ | 106/557 [00:00<00:00, 1456.39it/s] Saving model.layers.13.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 19%|█▉ | 107/557 [00:00<00:00, 1468.57it/s] Saving model.layers.13.linear_attn.conv1d.weight (f32, [8192,1,4]): 19%|█▉ | 108/557 [00:00<00:00, 1480.73it/s] Saving model.layers.13.linear_attn.conv1d.bias (f32, [8192]): 20%|█▉ | 109/557 [00:00<00:00, 1493.21it/s] Saving model.layers.13.linear_attn.dt_bias (f32, [32]): 20%|█▉ | 110/557 [00:00<00:00, 1505.81it/s] Saving model.layers.13.linear_attn.neg_exp_A (f32, [32]): 20%|█▉ | 111/557 [00:00<00:00, 1518.83it/s] Saving model.layers.13.linear_attn.norm.weight (f32, [128]): 20%|██ | 112/557 [00:00<00:00, 1531.73it/s] Saving model.layers.13.post_attention_layernorm.weight (f32, [2560]): 20%|██ | 113/557 [00:00<00:00, 1544.59it/s] Saving model.layers.14.input_layernorm.weight (f32, [2560]): 20%|██ | 114/557 [00:00<00:00, 1557.19it/s] Saving model.layers.14.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 21%|██ | 115/557 [00:00<00:00, 1569.90it/s] Saving model.layers.14.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 21%|██ | 116/557 [00:00<00:00, 1581.42it/s] Saving model.layers.14.linear_attn.conv1d.weight (f32, [8192,1,4]): 21%|██ | 117/557 [00:00<00:00, 1592.90it/s] Saving model.layers.14.linear_attn.conv1d.bias (f32, [8192]): 21%|██ | 118/557 [00:00<00:00, 1604.94it/s] Saving model.layers.14.linear_attn.dt_bias (f32, [32]): 21%|██▏ | 119/557 [00:00<00:00, 1617.37it/s] Saving model.layers.14.linear_attn.neg_exp_A (f32, [32]): 22%|██▏ | 120/557 [00:00<00:00, 1630.37it/s] Saving model.layers.14.linear_attn.norm.weight (f32, [128]): 22%|██▏ | 121/557 [00:00<00:00, 1643.34it/s] Saving model.layers.14.post_attention_layernorm.weight (f32, [2560]): 22%|██▏ | 122/557 [00:00<00:00, 1656.37it/s] Saving model.layers.15.input_layernorm.weight (f32, [2560]): 22%|██▏ | 123/557 [00:00<00:00, 1669.25it/s] Saving model.layers.15.attn.q_norm.layernorm.weight (f32, [256]): 22%|██▏ | 124/557 [00:00<00:00, 1682.13it/s] Saving model.layers.15.attn.k_norm.layernorm.weight (f32, [256]): 22%|██▏ | 125/557 [00:00<00:00, 1695.19it/s] Saving model.layers.15.post_attention_layernorm.weight (f32, [2560]): 23%|██▎ | 126/557 [00:00<00:00, 1708.17it/s] Saving model.layers.16.input_layernorm.weight (f32, [2560]): 23%|██▎ | 127/557 [00:00<00:00, 1720.97it/s] Saving model.layers.16.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 23%|██▎ | 128/557 [00:00<00:00, 1733.82it/s] Saving model.layers.16.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 23%|██▎ | 129/557 [00:00<00:00, 1745.69it/s] Saving model.layers.16.linear_attn.conv1d.weight (f32, [8192,1,4]): 23%|██▎ | 130/557 [00:00<00:00, 1757.21it/s] Saving model.layers.16.linear_attn.conv1d.bias (f32, [8192]): 24%|██▎ | 131/557 [00:00<00:00, 1769.17it/s] Saving model.layers.16.linear_attn.dt_bias (f32, [32]): 24%|██▎ | 132/557 [00:00<00:00, 1781.70it/s] Saving model.layers.16.linear_attn.neg_exp_A (f32, [32]): 24%|██▍ | 133/557 [00:00<00:00, 1794.68it/s] Saving model.layers.16.linear_attn.norm.weight (f32, [128]): 24%|██▍ | 134/557 [00:00<00:00, 1807.56it/s] Saving model.layers.16.post_attention_layernorm.weight (f32, [2560]): 24%|██▍ | 135/557 [00:00<00:00, 1820.40it/s] Saving model.layers.17.input_layernorm.weight (f32, [2560]): 24%|██▍ | 136/557 [00:00<00:00, 1833.04it/s] Saving model.layers.17.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 25%|██▍ | 137/557 [00:00<00:00, 1845.68it/s] Saving model.layers.17.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 25%|██▍ | 138/557 [00:00<00:00, 1857.04it/s] Saving model.layers.17.linear_attn.conv1d.weight (f32, [8192,1,4]): 25%|██▍ | 139/557 [00:00<00:00, 1868.13it/s] Saving model.layers.17.linear_attn.conv1d.bias (f32, [8192]): 25%|██▌ | 140/557 [00:00<00:00, 1879.82it/s] Saving model.layers.17.linear_attn.dt_bias (f32, [32]): 25%|██▌ | 141/557 [00:00<00:00, 1892.03it/s] Saving model.layers.17.linear_attn.neg_exp_A (f32, [32]): 25%|██▌ | 142/557 [00:00<00:00, 1904.89it/s] Saving model.layers.17.linear_attn.norm.weight (f32, [128]): 26%|██▌ | 143/557 [00:00<00:00, 1917.63it/s] Saving model.layers.17.post_attention_layernorm.weight (f32, [2560]): 26%|██▌ | 144/557 [00:00<00:00, 1930.43it/s] Saving model.layers.18.input_layernorm.weight (f32, [2560]): 26%|██▌ | 145/557 [00:00<00:00, 1942.88it/s] Saving model.layers.18.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 26%|██▌ | 146/557 [00:00<00:00, 1955.35it/s] Saving model.layers.18.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 26%|██▋ | 147/557 [00:00<00:00, 1966.49it/s] Saving model.layers.18.linear_attn.conv1d.weight (f32, [8192,1,4]): 27%|██▋ | 148/557 [00:00<00:00, 1977.82it/s] Saving model.layers.18.linear_attn.conv1d.bias (f32, [8192]): 27%|██▋ | 149/557 [00:00<00:00, 1989.79it/s] Saving model.layers.18.linear_attn.dt_bias (f32, [32]): 27%|██▋ | 150/557 [00:00<00:00, 2002.11it/s] Saving model.layers.18.linear_attn.neg_exp_A (f32, [32]): 27%|██▋ | 151/557 [00:00<00:00, 2014.86it/s] Saving model.layers.18.linear_attn.norm.weight (f32, [128]): 27%|██▋ | 152/557 [00:00<00:00, 2027.51it/s] Saving model.layers.18.post_attention_layernorm.weight (f32, [2560]): 27%|██▋ | 153/557 [00:00<00:00, 2040.21it/s] Saving model.layers.19.input_layernorm.weight (f32, [2560]): 28%|██▊ | 154/557 [00:00<00:00, 2052.67it/s] Saving model.layers.19.attn.q_norm.layernorm.weight (f32, [256]): 28%|██▊ | 155/557 [00:00<00:00, 2065.16it/s] Saving model.layers.19.attn.k_norm.layernorm.weight (f32, [256]): 28%|██▊ | 156/557 [00:00<00:00, 2077.88it/s] Saving model.layers.19.post_attention_layernorm.weight (f32, [2560]): 28%|██▊ | 157/557 [00:00<00:00, 2090.51it/s] Saving model.layers.20.input_layernorm.weight (f32, [2560]): 28%|██▊ | 158/557 [00:00<00:00, 2102.93it/s] Saving model.layers.20.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 29%|██▊ | 159/557 [00:00<00:00, 2115.26it/s] Saving model.layers.20.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 29%|██▊ | 160/557 [00:00<00:00, 2126.32it/s] Saving model.layers.20.linear_attn.conv1d.weight (f32, [8192,1,4]): 29%|██▉ | 161/557 [00:00<00:00, 2137.38it/s] Saving model.layers.20.linear_attn.conv1d.bias (f32, [8192]): 29%|██▉ | 162/557 [00:00<00:00, 2148.63it/s] Saving model.layers.20.linear_attn.dt_bias (f32, [32]): 29%|██▉ | 163/557 [00:00<00:00, 2160.67it/s] Saving model.layers.20.linear_attn.neg_exp_A (f32, [32]): 29%|██▉ | 164/557 [00:00<00:00, 2173.23it/s] Saving model.layers.20.linear_attn.norm.weight (f32, [128]): 30%|██▉ | 165/557 [00:00<00:00, 2185.76it/s] Saving model.layers.20.post_attention_layernorm.weight (f32, [2560]): 30%|██▉ | 166/557 [00:00<00:00, 2198.29it/s] Saving model.layers.21.input_layernorm.weight (f32, [2560]): 30%|██▉ | 167/557 [00:00<00:00, 2210.53it/s] Saving model.layers.21.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 30%|███ | 168/557 [00:00<00:00, 2222.87it/s] Saving model.layers.21.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 30%|███ | 169/557 [00:00<00:00, 2233.83it/s] Saving model.layers.21.linear_attn.conv1d.weight (f32, [8192,1,4]): 31%|███ | 170/557 [00:00<00:00, 2244.45it/s] Saving model.layers.21.linear_attn.conv1d.bias (f32, [8192]): 31%|███ | 171/557 [00:00<00:00, 2256.04it/s] Saving model.layers.21.linear_attn.dt_bias (f32, [32]): 31%|███ | 172/557 [00:00<00:00, 2268.09it/s] Saving model.layers.21.linear_attn.neg_exp_A (f32, [32]): 31%|███ | 173/557 [00:00<00:00, 2280.62it/s] Saving model.layers.21.linear_attn.norm.weight (f32, [128]): 31%|███ | 174/557 [00:00<00:00, 2293.05it/s] Saving model.layers.21.post_attention_layernorm.weight (f32, [2560]): 31%|███▏ | 175/557 [00:00<00:00, 2301.72it/s] Saving model.layers.22.input_layernorm.weight (f32, [2560]): 32%|███▏ | 176/557 [00:00<00:00, 2313.40it/s] Saving model.layers.22.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 32%|███▏ | 177/557 [00:00<00:00, 2325.49it/s] Saving model.layers.22.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 32%|███▏ | 178/557 [00:00<00:00, 2335.88it/s] Saving model.layers.22.linear_attn.conv1d.weight (f32, [8192,1,4]): 32%|███▏ | 179/557 [00:00<00:00, 2346.29it/s] Saving model.layers.22.linear_attn.conv1d.bias (f32, [8192]): 32%|███▏ | 180/557 [00:00<00:00, 2357.51it/s] Saving model.layers.22.linear_attn.dt_bias (f32, [32]): 32%|███▏ | 181/557 [00:00<00:00, 2369.26it/s] Saving model.layers.22.linear_attn.neg_exp_A (f32, [32]): 33%|███▎ | 182/557 [00:00<00:00, 2381.65it/s] Saving model.layers.22.linear_attn.norm.weight (f32, [128]): 33%|███▎ | 183/557 [00:00<00:00, 2393.97it/s] Saving model.layers.22.post_attention_layernorm.weight (f32, [2560]): 33%|███▎ | 184/557 [00:00<00:00, 2406.35it/s] Saving model.layers.23.input_layernorm.weight (f32, [2560]): 33%|███▎ | 185/557 [00:00<00:00, 2418.38it/s] Saving model.layers.23.attn.q_norm.layernorm.weight (f32, [256]): 33%|███▎ | 186/557 [00:00<00:00, 2430.47it/s] Saving model.layers.23.attn.k_norm.layernorm.weight (f32, [256]): 34%|███▎ | 187/557 [00:00<00:00, 2442.86it/s] Saving model.layers.23.post_attention_layernorm.weight (f32, [2560]): 34%|███▍ | 188/557 [00:00<00:00, 2455.15it/s] Saving model.layers.24.input_layernorm.weight (f32, [2560]): 34%|███▍ | 189/557 [00:00<00:00, 2467.12it/s] Saving model.layers.24.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 34%|███▍ | 190/557 [00:00<00:00, 2479.12it/s] Saving model.layers.24.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 34%|███▍ | 191/557 [00:00<00:00, 2489.53it/s] Saving model.layers.24.linear_attn.conv1d.weight (f32, [8192,1,4]): 34%|███▍ | 192/557 [00:00<00:00, 2500.07it/s] Saving model.layers.24.linear_attn.conv1d.bias (f32, [8192]): 35%|███▍ | 193/557 [00:00<00:00, 2511.22it/s] Saving model.layers.24.linear_attn.dt_bias (f32, [32]): 35%|███▍ | 194/557 [00:00<00:00, 2522.98it/s] Saving model.layers.24.linear_attn.neg_exp_A (f32, [32]): 35%|███▌ | 195/557 [00:00<00:00, 2535.23it/s] Saving model.layers.24.linear_attn.norm.weight (f32, [128]): 35%|███▌ | 196/557 [00:00<00:00, 2547.38it/s] Saving model.layers.24.post_attention_layernorm.weight (f32, [2560]): 35%|███▌ | 197/557 [00:00<00:00, 2559.63it/s] Saving model.layers.25.input_layernorm.weight (f32, [2560]): 36%|███▌ | 198/557 [00:00<00:00, 2571.47it/s] Saving model.layers.25.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 36%|███▌ | 199/557 [00:00<00:00, 2583.46it/s] Saving model.layers.25.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 36%|███▌ | 200/557 [00:00<00:00, 2593.50it/s] Saving model.layers.25.linear_attn.conv1d.weight (f32, [8192,1,4]): 36%|███▌ | 201/557 [00:00<00:00, 2603.60it/s] Saving model.layers.25.linear_attn.conv1d.bias (f32, [8192]): 36%|███▋ | 202/557 [00:00<00:00, 2614.68it/s] Saving model.layers.25.linear_attn.dt_bias (f32, [32]): 36%|███▋ | 203/557 [00:00<00:00, 2626.40it/s] Saving model.layers.25.linear_attn.neg_exp_A (f32, [32]): 37%|███▋ | 204/557 [00:00<00:00, 2638.63it/s] Saving model.layers.25.linear_attn.norm.weight (f32, [128]): 37%|███▋ | 205/557 [00:00<00:00, 2650.77it/s] Saving model.layers.25.post_attention_layernorm.weight (f32, [2560]): 37%|███▋ | 206/557 [00:00<00:00, 2662.87it/s] Saving model.layers.26.input_layernorm.weight (f32, [2560]): 37%|███▋ | 207/557 [00:00<00:00, 2674.42it/s] Saving model.layers.26.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 37%|███▋ | 208/557 [00:00<00:00, 2686.23it/s] Saving model.layers.26.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 38%|███▊ | 209/557 [00:00<00:00, 2696.66it/s] Saving model.layers.26.linear_attn.conv1d.weight (f32, [8192,1,4]): 38%|███▊ | 210/557 [00:00<00:00, 2707.02it/s] Saving model.layers.26.linear_attn.conv1d.bias (f32, [8192]): 38%|███▊ | 211/557 [00:00<00:00, 2718.25it/s] Saving model.layers.26.linear_attn.dt_bias (f32, [32]): 38%|███▊ | 212/557 [00:00<00:00, 2729.90it/s] Saving model.layers.26.linear_attn.neg_exp_A (f32, [32]): 38%|███▊ | 213/557 [00:00<00:00, 2742.07it/s] Saving model.layers.26.linear_attn.norm.weight (f32, [128]): 38%|███▊ | 214/557 [00:00<00:00, 2754.09it/s] Saving model.layers.26.post_attention_layernorm.weight (f32, [2560]): 39%|███▊ | 215/557 [00:00<00:00, 2766.17it/s] Saving model.layers.27.input_layernorm.weight (f32, [2560]): 39%|███▉ | 216/557 [00:00<00:00, 2778.00it/s] Saving model.layers.27.attn.q_norm.layernorm.weight (f32, [256]): 39%|███▉ | 217/557 [00:00<00:00, 2789.83it/s] Saving model.layers.27.attn.k_norm.layernorm.weight (f32, [256]): 39%|███▉ | 218/557 [00:00<00:00, 2801.96it/s] Saving model.layers.27.post_attention_layernorm.weight (f32, [2560]): 39%|███▉ | 219/557 [00:00<00:00, 2813.96it/s] Saving model.layers.28.input_layernorm.weight (f32, [2560]): 39%|███▉ | 220/557 [00:00<00:00, 2825.72it/s] Saving model.layers.28.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 40%|███▉ | 221/557 [00:00<00:00, 2837.40it/s] Saving model.layers.28.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 40%|███▉ | 222/557 [00:00<00:00, 2847.75it/s] Saving model.layers.28.linear_attn.conv1d.weight (f32, [8192,1,4]): 40%|████ | 223/557 [00:00<00:00, 2857.76it/s] Saving model.layers.28.linear_attn.conv1d.bias (f32, [8192]): 40%|████ | 224/557 [00:00<00:00, 2868.74it/s] Saving model.layers.28.linear_attn.dt_bias (f32, [32]): 40%|████ | 225/557 [00:00<00:00, 2880.25it/s] Saving model.layers.28.linear_attn.neg_exp_A (f32, [32]): 41%|████ | 226/557 [00:00<00:00, 2892.27it/s] Saving model.layers.28.linear_attn.norm.weight (f32, [128]): 41%|████ | 227/557 [00:00<00:00, 2904.14it/s] Saving model.layers.28.post_attention_layernorm.weight (f32, [2560]): 41%|████ | 228/557 [00:00<00:00, 2915.92it/s] Saving model.layers.29.input_layernorm.weight (f32, [2560]): 41%|████ | 229/557 [00:00<00:00, 2927.52it/s] Saving model.layers.29.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 41%|████▏ | 230/557 [00:00<00:00, 2939.25it/s] Saving model.layers.29.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 41%|████▏ | 231/557 [00:00<00:00, 2949.31it/s] Saving model.layers.29.linear_attn.conv1d.weight (f32, [8192,1,4]): 42%|████▏ | 232/557 [00:00<00:00, 2958.24it/s] Saving model.layers.29.linear_attn.conv1d.bias (f32, [8192]): 42%|████▏ | 233/557 [00:00<00:00, 2968.86it/s] Saving model.layers.29.linear_attn.dt_bias (f32, [32]): 42%|████▏ | 234/557 [00:00<00:00, 2980.25it/s] Saving model.layers.29.linear_attn.neg_exp_A (f32, [32]): 42%|████▏ | 235/557 [00:00<00:00, 2992.18it/s] Saving model.layers.29.linear_attn.norm.weight (f32, [128]): 42%|████▏ | 236/557 [00:00<00:00, 3004.03it/s] Saving model.layers.29.post_attention_layernorm.weight (f32, [2560]): 43%|████▎ | 237/557 [00:00<00:00, 3015.93it/s] Saving model.layers.30.input_layernorm.weight (f32, [2560]): 43%|████▎ | 238/557 [00:00<00:00, 3027.53it/s] Saving model.layers.30.linear_attn.b_proj.MatMul.weight (f32, [2560,32]): 43%|████▎ | 239/557 [00:00<00:00, 3039.17it/s] Saving model.layers.30.linear_attn.a_proj.MatMul.weight (f32, [2560,32]): 43%|████▎ | 240/557 [00:00<00:00, 3048.94it/s] Saving model.layers.30.linear_attn.conv1d.weight (f32, [8192,1,4]): 43%|████▎ | 241/557 [00:00<00:00, 3058.55it/s] Saving model.layers.30.linear_attn.conv1d.bias (f32, [8192]): 43%|████▎ | 242/557 [00:00<00:00, 3069.38it/s] Saving model.layers.30.linear_attn.dt_bias (f32, [32]): 44%|████▎ | 243/557 [00:00<00:00, 3080.65it/s] Saving model.layers.30.linear_attn.neg_exp_A (f32, [32]): 44%|████▍ | 244/557 [00:00<00:00, 3092.54it/s] Saving model.layers.30.linear_attn.norm.weight (f32, [128]): 44%|████▍ | 245/557 [00:00<00:00, 3104.31it/s] Saving model.layers.30.post_attention_layernorm.weight (f32, [2560]): 44%|████▍ | 246/557 [00:00<00:00, 3116.09it/s] Saving model.layers.31.input_layernorm.weight (f32, [2560]): 44%|████▍ | 247/557 [00:00<00:00, 3127.58it/s] Saving model.layers.31.attn.q_norm.layernorm.weight (f32, [256]): 45%|████▍ | 248/557 [00:00<00:00, 3139.13it/s] Saving model.layers.31.attn.k_norm.layernorm.weight (f32, [256]): 45%|████▍ | 249/557 [00:00<00:00, 3150.95it/s] Saving model.layers.31.post_attention_layernorm.weight (f32, [2560]): 45%|████▍ | 250/557 [00:00<00:00, 3162.60it/s] Saving model.layers.32.final_norm_layernorm.weight (f32, [2560]): 45%|████▌ | 251/557 [00:00<00:00, 3174.11it/s] Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 45%|████▌ | 252/557 [00:00<00:00, 3185.61it/s] Saving model.layers.0.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 45%|████▌ | 253/557 [00:00<00:00, 3142.49it/s] Saving model.layers.0.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 46%|████▌ | 254/557 [00:00<00:00, 3141.36it/s] Saving model.layers.0.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 46%|████▌ | 255/557 [00:00<00:00, 3118.58it/s] Saving model.layers.0.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 46%|████▌ | 256/557 [00:00<00:00, 3110.02it/s] Saving model.layers.0.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 46%|████▌ | 257/557 [00:00<00:00, 3097.67it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 46%|████▋ | 258/557 [00:00<00:00, 3102.85it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 46%|████▋ | 259/557 [00:00<00:00, 2989.82it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 47%|████▋ | 260/557 [00:00<00:00, 2964.63it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 47%|████▋ | 261/557 [00:00<00:00, 2918.34it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 47%|████▋ | 262/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 47%|████▋ | 262/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 47%|████▋ | 263/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 47%|████▋ | 264/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 48%|████▊ | 265/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 48%|████▊ | 266/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 48%|████▊ | 267/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 48%|████▊ | 268/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 48%|████▊ | 269/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 48%|████▊ | 270/557 [00:00<00:00, 2292.53it/s] Saving model.layers.1.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 49%|████▊ | 271/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 49%|████▉ | 272/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 49%|████▉ | 273/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 49%|████▉ | 274/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 49%|████▉ | 275/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 50%|████▉ | 276/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 50%|████▉ | 277/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 50%|████▉ | 278/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 50%|█████ | 279/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 50%|█████ | 280/557 [00:00<00:00, 2292.53it/s] Saving model.layers.2.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 50%|█████ | 281/557 [00:00<00:00, 2292.53it/s] Saving model.layers.3.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 51%|█████ | 282/557 [00:00<00:00, 2292.53it/s] Saving model.layers.3.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 51%|█████ | 283/557 [00:00<00:00, 2292.53it/s] Saving model.layers.3.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 51%|█████ | 284/557 [00:00<00:00, 2292.53it/s] Saving model.layers.3.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 51%|█████ | 285/557 [00:00<00:00, 2292.53it/s] Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 51%|█████▏ | 286/557 [00:00<00:00, 2292.53it/s] Saving model.layers.3.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 52%|█████▏ | 287/557 [00:00<00:00, 2292.53it/s] Saving model.layers.3.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 52%|█████▏ | 288/557 [00:00<00:00, 2292.53it/s] Saving model.layers.3.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 52%|█████▏ | 289/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 52%|█████▏ | 290/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 52%|█████▏ | 291/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 52%|█████▏ | 292/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 53%|█████▎ | 293/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 53%|█████▎ | 294/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 53%|█████▎ | 295/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 53%|█████▎ | 296/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 53%|█████▎ | 297/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 54%|█████▎ | 298/557 [00:00<00:00, 2292.53it/s] Saving model.layers.4.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 54%|█████▎ | 299/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 54%|█████▍ | 300/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 54%|█████▍ | 301/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 54%|█████▍ | 302/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 54%|█████▍ | 303/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 55%|█████▍ | 304/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 55%|█████▍ | 305/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 55%|█████▍ | 306/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 55%|█████▌ | 307/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 55%|█████▌ | 308/557 [00:00<00:00, 2292.53it/s] Saving model.layers.5.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 55%|█████▌ | 309/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 56%|█████▌ | 310/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 56%|█████▌ | 311/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 56%|█████▌ | 312/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 56%|█████▌ | 313/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 56%|█████▋ | 314/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 57%|█████▋ | 315/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 57%|█████▋ | 316/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 57%|█████▋ | 317/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 57%|█████▋ | 318/557 [00:00<00:00, 2292.53it/s] Saving model.layers.6.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 57%|█████▋ | 319/557 [00:00<00:00, 2292.53it/s] Saving model.layers.7.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 57%|█████▋ | 320/557 [00:00<00:00, 2292.53it/s] Saving model.layers.7.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 58%|█████▊ | 321/557 [00:00<00:00, 2292.53it/s] Saving model.layers.7.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 58%|█████▊ | 322/557 [00:00<00:00, 2292.53it/s] Saving model.layers.7.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 58%|█████▊ | 323/557 [00:00<00:00, 2292.53it/s] Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 58%|█████▊ | 324/557 [00:00<00:00, 2292.53it/s] Saving model.layers.7.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 58%|█████▊ | 325/557 [00:00<00:00, 2292.53it/s] Saving model.layers.7.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 59%|█████▊ | 326/557 [00:00<00:00, 2292.53it/s] Saving model.layers.7.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 59%|█████▊ | 327/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 59%|█████▉ | 328/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 59%|█████▉ | 329/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 59%|█████▉ | 330/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 59%|█████▉ | 331/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 60%|█████▉ | 332/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 60%|█████▉ | 333/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 60%|█████▉ | 334/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 60%|██████ | 335/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 60%|██████ | 336/557 [00:00<00:00, 2292.53it/s] Saving model.layers.8.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 61%|██████ | 337/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 61%|██████ | 338/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 61%|██████ | 339/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 61%|██████ | 340/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 61%|██████ | 341/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 61%|██████▏ | 342/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 62%|██████▏ | 343/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 62%|██████▏ | 344/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 62%|██████▏ | 345/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 62%|██████▏ | 346/557 [00:00<00:00, 2292.53it/s] Saving model.layers.9.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 62%|██████▏ | 347/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 62%|██████▏ | 348/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 63%|██████▎ | 349/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 63%|██████▎ | 350/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 63%|██████▎ | 351/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 63%|██████▎ | 352/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 63%|██████▎ | 353/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 64%|██████▎ | 354/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 64%|██████▎ | 355/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 64%|██████▍ | 356/557 [00:00<00:00, 2292.53it/s] Saving model.layers.10.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 64%|██████▍ | 357/557 [00:00<00:00, 2292.53it/s] Saving model.layers.11.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 64%|██████▍ | 358/557 [00:00<00:00, 2292.53it/s] Saving model.layers.11.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 64%|██████▍ | 359/557 [00:00<00:00, 2292.53it/s] Saving model.layers.11.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 65%|██████▍ | 360/557 [00:00<00:00, 2292.53it/s] Saving model.layers.11.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 65%|██████▍ | 361/557 [00:00<00:00, 2292.53it/s] Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 65%|██████▍ | 362/557 [00:00<00:00, 2292.53it/s] Saving model.layers.11.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 65%|██████▌ | 363/557 [00:00<00:00, 2292.53it/s] Saving model.layers.11.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 65%|██████▌ | 364/557 [00:00<00:00, 2292.53it/s] Saving model.layers.11.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 66%|██████▌ | 365/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 66%|██████▌ | 366/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 66%|██████▌ | 367/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 66%|██████▌ | 368/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 66%|██████▌ | 369/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 66%|██████▋ | 370/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 67%|██████▋ | 371/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 67%|██████▋ | 372/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 67%|██████▋ | 373/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 67%|██████▋ | 374/557 [00:00<00:00, 2292.53it/s] Saving model.layers.12.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 67%|██████▋ | 375/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 68%|██████▊ | 376/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 68%|██████▊ | 377/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 68%|██████▊ | 378/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 68%|██████▊ | 379/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 68%|██████▊ | 380/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 68%|██████▊ | 381/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 69%|██████▊ | 382/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 69%|██████▉ | 383/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 69%|██████▉ | 384/557 [00:00<00:00, 2292.53it/s] Saving model.layers.13.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 69%|██████▉ | 385/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 69%|██████▉ | 386/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 69%|██████▉ | 387/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 70%|██████▉ | 388/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 70%|██████▉ | 389/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 70%|███████ | 390/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 70%|███████ | 391/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 70%|███████ | 392/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 71%|███████ | 393/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 71%|███████ | 394/557 [00:00<00:00, 2292.53it/s] Saving model.layers.14.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 71%|███████ | 395/557 [00:00<00:00, 2292.53it/s] Saving model.layers.15.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 71%|███████ | 396/557 [00:00<00:00, 2292.53it/s] Saving model.layers.15.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 71%|███████▏ | 397/557 [00:00<00:00, 2292.53it/s] Saving model.layers.15.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 71%|███████▏ | 398/557 [00:00<00:00, 2292.53it/s] Saving model.layers.15.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 72%|███████▏ | 399/557 [00:00<00:00, 2292.53it/s] Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 72%|███████▏ | 400/557 [00:00<00:00, 2292.53it/s] Saving model.layers.15.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 72%|███████▏ | 401/557 [00:00<00:00, 2292.53it/s] Saving model.layers.15.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 72%|███████▏ | 402/557 [00:00<00:00, 2292.53it/s] Saving model.layers.15.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 72%|███████▏ | 403/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 73%|███████▎ | 404/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 73%|███████▎ | 405/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 73%|███████▎ | 406/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 73%|███████▎ | 407/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 73%|███████▎ | 408/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 73%|███████▎ | 409/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 74%|███████▎ | 410/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 74%|███████▍ | 411/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 74%|███████▍ | 412/557 [00:00<00:00, 2292.53it/s] Saving model.layers.16.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 74%|███████▍ | 413/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 74%|███████▍ | 414/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 75%|███████▍ | 415/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 75%|███████▍ | 416/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 75%|███████▍ | 417/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 75%|███████▌ | 418/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 75%|███████▌ | 419/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 75%|███████▌ | 420/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 76%|███████▌ | 421/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 76%|███████▌ | 422/557 [00:00<00:00, 2292.53it/s] Saving model.layers.17.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 76%|███████▌ | 423/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 76%|███████▌ | 424/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 76%|███████▋ | 425/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 76%|███████▋ | 426/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 77%|███████▋ | 427/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 77%|███████▋ | 428/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 77%|███████▋ | 429/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 77%|███████▋ | 430/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 77%|███████▋ | 431/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 78%|███████▊ | 432/557 [00:00<00:00, 2292.53it/s] Saving model.layers.18.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 78%|███████▊ | 433/557 [00:00<00:00, 2292.53it/s] Saving model.layers.19.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 78%|███████▊ | 434/557 [00:00<00:00, 2292.53it/s] Saving model.layers.19.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 78%|███████▊ | 435/557 [00:00<00:00, 2292.53it/s] Saving model.layers.19.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 78%|███████▊ | 436/557 [00:00<00:00, 2292.53it/s] Saving model.layers.19.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 78%|███████▊ | 437/557 [00:00<00:00, 2292.53it/s] Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 79%|███████▊ | 438/557 [00:00<00:00, 2292.53it/s] Saving model.layers.19.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 79%|███████▉ | 439/557 [00:00<00:00, 2292.53it/s] Saving model.layers.19.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 79%|███████▉ | 440/557 [00:00<00:00, 2292.53it/s] Saving model.layers.19.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 79%|███████▉ | 441/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 79%|███████▉ | 442/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 80%|███████▉ | 443/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 80%|███████▉ | 444/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 80%|███████▉ | 445/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 80%|████████ | 446/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 80%|████████ | 447/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 80%|████████ | 448/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 81%|████████ | 449/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 81%|████████ | 450/557 [00:00<00:00, 2292.53it/s] Saving model.layers.20.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 81%|████████ | 451/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 81%|████████ | 452/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 81%|████████▏ | 453/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 82%|████████▏ | 454/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 82%|████████▏ | 455/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 82%|████████▏ | 456/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 82%|████████▏ | 457/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 82%|████████▏ | 458/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 82%|████████▏ | 459/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 83%|████████▎ | 460/557 [00:00<00:00, 2292.53it/s] Saving model.layers.21.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 83%|████████▎ | 461/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 83%|████████▎ | 462/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 83%|████████▎ | 463/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 83%|████████▎ | 464/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 83%|████████▎ | 465/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 84%|████████▎ | 466/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 84%|████████▍ | 467/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 84%|████████▍ | 468/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 84%|████████▍ | 469/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 84%|████████▍ | 470/557 [00:00<00:00, 2292.53it/s] Saving model.layers.22.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 85%|████████▍ | 471/557 [00:00<00:00, 2292.53it/s] Saving model.layers.23.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 85%|████████▍ | 472/557 [00:00<00:00, 2292.53it/s] Saving model.layers.23.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 85%|████████▍ | 473/557 [00:00<00:00, 2292.53it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 85%|████████▌ | 474/557 [00:00<00:00, 2292.53it/s] Saving model.layers.23.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 85%|████████▌ | 475/557 [00:00<00:00, 2292.53it/s] Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 85%|████████▌ | 476/557 [00:00<00:00, 2292.53it/s] Saving model.layers.23.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 86%|████████▌ | 477/557 [00:00<00:00, 2292.53it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 86%|████████▌ | 478/557 [00:00<00:00, 2292.53it/s] Saving model.layers.23.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 86%|████████▌ | 479/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 86%|████████▌ | 480/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 86%|████████▋ | 481/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 87%|████████▋ | 482/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 87%|████████▋ | 483/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 87%|████████▋ | 484/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 87%|████████▋ | 485/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 87%|████████▋ | 486/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 87%|████████▋ | 487/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 88%|████████▊ | 488/557 [00:00<00:00, 2292.53it/s] Saving model.layers.24.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 88%|████████▊ | 489/557 [00:00<00:00, 2292.53it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 88%|████████▊ | 490/557 [00:00<00:00, 2292.53it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 88%|████████▊ | 491/557 [00:00<00:00, 2292.53it/s] Saving model.layers.25.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 88%|████████▊ | 492/557 [00:00<00:00, 608.59it/s] Saving model.layers.25.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 88%|████████▊ | 492/557 [00:00<00:00, 608.59it/s] Saving model.layers.25.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 89%|████████▊ | 493/557 [00:00<00:00, 608.59it/s] Saving model.layers.25.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 89%|████████▊ | 494/557 [00:00<00:00, 608.59it/s] Saving model.layers.25.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 89%|████████▉ | 495/557 [00:00<00:00, 608.59it/s] Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 89%|████████▉ | 496/557 [00:00<00:00, 608.59it/s] Saving model.layers.25.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 89%|████████▉ | 497/557 [00:00<00:00, 608.59it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 89%|████████▉ | 498/557 [00:00<00:00, 608.59it/s] Saving model.layers.25.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 90%|████████▉ | 499/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 90%|████████▉ | 500/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 90%|████████▉ | 501/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 90%|█████████ | 502/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 90%|█████████ | 503/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 90%|█████████ | 504/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 91%|█████████ | 505/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 91%|█████████ | 506/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 91%|█████████ | 507/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 91%|█████████ | 508/557 [00:00<00:00, 608.59it/s] Saving model.layers.26.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 91%|█████████▏| 509/557 [00:00<00:00, 608.59it/s] Saving model.layers.27.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 92%|█████████▏| 510/557 [00:00<00:00, 608.59it/s] Saving model.layers.27.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 92%|█████████▏| 511/557 [00:00<00:00, 608.59it/s] Saving model.layers.27.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 92%|█████████▏| 512/557 [00:00<00:00, 608.59it/s] Saving model.layers.27.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 92%|█████████▏| 513/557 [00:00<00:00, 608.59it/s] Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 92%|█████████▏| 514/557 [00:00<00:00, 608.59it/s] Saving model.layers.27.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 92%|█████████▏| 515/557 [00:00<00:00, 608.59it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 93%|█████████▎| 516/557 [00:00<00:00, 608.59it/s] Saving model.layers.27.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 93%|█████████▎| 517/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 93%|█████████▎| 518/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 93%|█████████▎| 519/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 93%|█████████▎| 520/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 94%|█████████▎| 521/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 94%|█████████▎| 522/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 94%|█████████▍| 523/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 94%|█████████▍| 524/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 94%|█████████▍| 525/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 94%|█████████▍| 526/557 [00:00<00:00, 608.59it/s] Saving model.layers.28.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 95%|█████████▍| 527/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 95%|█████████▍| 528/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 95%|█████████▍| 529/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 95%|█████████▌| 530/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 95%|█████████▌| 531/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 96%|█████████▌| 532/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 96%|█████████▌| 533/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 96%|█████████▌| 534/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 96%|█████████▌| 535/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 96%|█████████▌| 536/557 [00:00<00:00, 608.59it/s] Saving model.layers.29.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 96%|█████████▋| 537/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_Q4 (u8, [8192,80,16]): 97%|█████████▋| 538/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.linear_attn.qkv_proj.MatMul.weight_scales (f32, [8192,80]): 97%|█████████▋| 539/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.linear_attn.z_proj.MatMul.weight_Q4 (u8, [4096,80,16]): 97%|█████████▋| 540/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.linear_attn.z_proj.MatMul.weight_scales (f32, [4096,80]): 97%|█████████▋| 541/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.linear_attn.out_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 97%|█████████▋| 542/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.linear_attn.out_proj.MatMul.weight_scales (f32, [2560,128]): 97%|█████████▋| 543/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 98%|█████████▊| 544/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 98%|█████████▊| 545/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 98%|█████████▊| 546/557 [00:00<00:00, 608.59it/s] Saving model.layers.30.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 98%|█████████▊| 547/557 [00:00<00:00, 608.59it/s] Saving model.layers.31.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 98%|█████████▊| 548/557 [00:00<00:00, 608.59it/s] Saving model.layers.31.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 99%|█████████▊| 549/557 [00:00<00:00, 608.59it/s] Saving model.layers.31.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 99%|█████████▊| 550/557 [00:00<00:00, 608.59it/s] Saving model.layers.31.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 99%|█████████▉| 551/557 [00:00<00:00, 608.59it/s] Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 99%|█████████▉| 552/557 [00:00<00:00, 608.59it/s] Saving model.layers.31.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 99%|█████████▉| 553/557 [00:00<00:00, 608.59it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 99%|█████████▉| 554/557 [00:00<00:00, 608.59it/s] Saving model.layers.31.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 100%|█████████▉| 555/557 [00:00<00:00, 608.59it/s] Saving lm_head.MatMul.weight_Q4 (u8, [248320,80,16]): 100%|█████████▉| 556/557 [00:00<00:00, 608.59it/s] Saving lm_head.MatMul.weight_scales (f32, [248320,80]): 100%|██████████| 557/557 [00:01<00:00, 608.59it/s] Saving lm_head.MatMul.weight_scales (f32, [248320,80]): 100%|██████████| 557/557 [00:01<00:00, 503.27it/s] 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[1] ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/attn_mask_reformat/attn_mask_subgraph/ReduceSum ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/attn_mask_reformat/attn_mask_subgraph/Sub ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/attn_mask_reformat/attn_mask_subgraph/Sub/Cast ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/attn_mask_reformat/attn_mask_subgraph/Shape ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/1 ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize /model/attn_mask_reformat/attn_mask_subgraph/Gather ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - Gather doesn't have const weight. Skip quantization. 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of /model/attn_mask_reformat/attn_mask_subgraph/Gather with 4 bits ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/attn_mask_reformat/attn_mask_subgraph/Gather/Cast ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[248320, 1280] ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/mtp/embed_tokens/Reshape ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/mtp/embed_tokens/GatherBlockQuantized ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/mtp/pre_fc_norm_embedding ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/mtp/pre_fc_norm_hidden ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/mtp/fc/Concat ... 2026-11-05 03:43:03,790 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize /model/mtp/fc/MatMul ... 2026-11-05 03:43:03,823 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of /model/mtp/fc/MatMul with 4 bits ... 2026-11-05 03:43:03,823 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/input_layernorm/LayerNorm ... 2026-11-05 03:43:03,823 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize /model/layers.0/attn/qkv_proj/MatMul ... 2026-11-05 03:43:03,870 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of /model/layers.0/attn/qkv_proj/MatMul with 4 bits ... 2026-11-05 03:43:03,870 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[8192, 1024, 1024] ... 2026-11-05 03:43:03,870 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/qkv_proj/Split ... 2026-11-05 03:43:03,870 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[0, 0, 16, 512] ... 2026-11-05 03:43:03,870 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/q_gate/Reshape ... 2026-11-05 03:43:03,870 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[256, 256] ... 2026-11-05 03:43:03,870 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/q_gate/Split ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[0, 0, 4096] ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/q_proj/Reshape ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/gate/Reshape ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[0, -1, 256] ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/q_norm/Reshape_1 ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/q_norm/SimplifiedLayerNormalization ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[0, -1, 4096] ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/q_norm/Reshape_2 ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/k_norm/Reshape_1 ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/k_norm/SimplifiedLayerNormalization ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/constant_nodes/INT64/[0, -1, 1024] ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/k_norm/Reshape_2 ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/q_rotary/MRotaryEmbedding ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/k_rotary/MRotaryEmbedding ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/GroupQueryAttention ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/gate/Sigmoid ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/attn/gate/Mul ... 2026-11-05 03:43:03,871 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize /model/layers.0/attn/o_proj/MatMul ... 2026-11-05 03:43:03,896 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of /model/layers.0/attn/o_proj/MatMul with 4 bits ... 2026-11-05 03:43:03,896 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/post_attention_layernorm/SkipLayerNorm ... 2026-11-05 03:43:03,896 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize /model/layers.0/mlp/gate_up_proj/MatMul ... 2026-11-05 03:43:04,022 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of /model/layers.0/mlp/gate_up_proj/MatMul with 4 bits ... 2026-11-05 03:43:04,022 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/mlp/gate_up_proj/Split ... 2026-11-05 03:43:04,022 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/mlp/act_fn/Sigmoid ... 2026-11-05 03:43:04,022 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/mlp/act_fn/Mul ... 2026-11-05 03:43:04,022 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.0/mlp/Mul ... 2026-11-05 03:43:04,022 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize /model/layers.0/mlp/down_proj/MatMul ... 2026-11-05 03:43:04,070 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of /model/layers.0/mlp/down_proj/MatMul with 4 bits ... 2026-11-05 03:43:04,070 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - skip to quantize /model/layers.1/final_norm_layernorm/SkipLayerNorm ... 2026-11-05 03:43:04,070 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - start to quantize /lm_head/MatMul ... 2026-11-05 03:43:06,450 onnxruntime.quantization.matmul_nbits_quantizer [INFO] - complete quantization of /lm_head/MatMul with 4 bits ... Saving ONNX model in /home/anon/knowline/onnx/model-cpu 0it [00:00, ?it/s] Saving model.mtp.pre_fc_norm_embedding.weight (f32, [2560]): 5%|▍ | 1/21 [00:00<00:00, 735.46it/s] Saving model.mtp.pre_fc_norm_hidden.weight (f32, [2560]): 10%|▉ | 2/21 [00:00<00:00, 1372.71it/s] Saving model.layers.0.input_layernorm.weight (f32, [2560]): 14%|█▍ | 3/21 [00:00<00:00, 1997.92it/s] Saving model.layers.0.attn.q_norm.layernorm.weight (f32, [256]): 19%|█▉ | 4/21 [00:00<00:00, 2603.94it/s] Saving model.layers.0.attn.k_norm.layernorm.weight (f32, [256]): 24%|██▍ | 5/21 [00:00<00:00, 3199.80it/s] Saving cos_cache (f32, [262144,32]): 29%|██▊ | 6/21 [00:00<00:00, 3772.99it/s] Saving sin_cache (f32, [262144,32]): 33%|███▎ | 7/21 [00:00<00:00, 1170.66it/s] Saving model.layers.0.post_attention_layernorm.weight (f32, [2560]): 38%|███▊ | 8/21 [00:00<00:00, 758.81it/s] Saving model.layers.1.final_norm_layernorm.weight (f32, [2560]): 43%|████▎ | 9/21 [00:00<00:00, 845.23it/s] Saving model.mtp.fc.MatMul.weight_Q4 (u8, [2560,160,16]): 48%|████▊ | 10/21 [00:00<00:00, 935.04it/s] Saving model.mtp.fc.MatMul.weight_scales (f32, [2560,160]): 52%|█████▏ | 11/21 [00:00<00:00, 953.27it/s] Saving model.layers.0.attn.qkv_proj.MatMul.weight_Q4 (u8, [10240,80,16]): 57%|█████▋ | 12/21 [00:00<00:00, 1019.48it/s] Saving model.layers.0.attn.qkv_proj.MatMul.weight_scales (f32, [10240,80]): 62%|██████▏ | 13/21 [00:00<00:00, 961.08it/s] Saving model.layers.0.attn.o_proj.MatMul.weight_Q4 (u8, [2560,128,16]): 67%|██████▋ | 14/21 [00:00<00:00, 986.23it/s] Saving model.layers.0.attn.o_proj.MatMul.weight_scales (f32, [2560,128]): 71%|███████▏ | 15/21 [00:00<00:00, 1003.15it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_Q4 (u8, [18432,80,16]): 76%|███████▌ | 16/21 [00:00<00:00, 1055.02it/s] Saving model.layers.0.mlp.gate_up_proj.MatMul.input_1_scales (f32, [18432,80]): 81%|████████ | 17/21 [00:00<00:00, 932.48it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_Q4 (u8, [2560,288,16]): 86%|████████▌ | 18/21 [00:00<00:00, 946.45it/s] Saving model.layers.0.mlp.down_proj.MatMul.weight_scales (f32, [2560,288]): 90%|█████████ | 19/21 [00:00<00:00, 915.77it/s] Saving lm_head.MatMul.weight_Q4 (u8, [248320,80,16]): 95%|█████████▌| 20/21 [00:00<00:00, 945.72it/s] Saving lm_head.MatMul.weight_Q4 (u8, [248320,80,16]): 100%|██████████| 21/21 [00:00<00:00, 131.28it/s] Saving lm_head.MatMul.weight_scales (f32, [248320,80]): 100%|██████████| 21/21 [00:00<00:00, 131.28it/s] Saving lm_head.MatMul.weight_scales (f32, [248320,80]): 100%|██████████| 21/21 [00:00<00:00, 114.16it/s] Configured 397 MB of runtime initializer sharing; /home/anon/knowline/onnx/model-cpu/mtp.onnx.data remains self-contained. Warning: could not read generation_config.json (/home/anon/LLM/KnowLine-4B-Gen5.5 does not appear to have a file named generation_config.json. Checkout 'https://huggingface.co//home/anon/LLM/KnowLine-4B-Gen5.5/tree/main' for available files.). Falling back to config.json. Adding the tokenizer's end-of-turn token <|im_end|> (id 248046) to eos_token_id from the model configuration: 248044. Saving GenAI config in /home/anon/knowline/onnx/model-cpu Added 'mtp' section to genai_config.json Saving processing files in /home/anon/knowline/onnx/model-cpu for GenAI EXIT:0