File size: 156,749 Bytes
598b018 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Loading teacher from /home/void0x14/.cache/huggingface/hub/models--Qwen--Qwen3.5-0.8B-Base/snapshots/dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68...
Loading weights: 0%| | 0/473 [00:00<?, ?it/s]
Loading weights: 0%| | 1/473 [00:00<00:00, 7371.36it/s, Materializing param=language_model.embed_tokens.weight]
Loading weights: 0%| | 1/473 [00:00<00:00, 3130.08it/s, Materializing param=language_model.embed_tokens.weight]
Loading weights: 0%| | 2/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.embed_tokens.weight]
Loading weights: 0%| | 2/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.layers.0.input_layernorm.weight]
Loading weights: 0%| | 2/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.layers.0.input_layernorm.weight]
Loading weights: 1%| | 3/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.A_log]
Loading weights: 1%| | 3/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.A_log]
Loading weights: 1%| | 4/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.conv1d.weight]
Loading weights: 1%| | 4/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.conv1d.weight]
Loading weights: 1%| | 5/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.dt_bias]
Loading weights: 1%| | 5/473 [00:00<00:59, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.dt_bias]
Loading weights: 1%|▏ | 6/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.in_proj_a.weight]
Loading weights: 1%|▏ | 6/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.in_proj_a.weight]
Loading weights: 1%|▏ | 7/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.in_proj_b.weight]
Loading weights: 1%|▏ | 7/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.in_proj_b.weight]
Loading weights: 2%|▏ | 8/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.in_proj_qkv.weight]
Loading weights: 2%|▏ | 8/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.in_proj_qkv.weight]
Loading weights: 2%|▏ | 9/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.in_proj_z.weight]
Loading weights: 2%|▏ | 9/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.in_proj_z.weight]
Loading weights: 2%|▏ | 10/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.norm.weight]
Loading weights: 2%|▏ | 10/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.norm.weight]
Loading weights: 2%|▏ | 11/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.out_proj.weight]
Loading weights: 2%|▏ | 11/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.linear_attn.out_proj.weight]
Loading weights: 3%|▎ | 12/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.mlp.down_proj.weight]
Loading weights: 3%|▎ | 12/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.mlp.down_proj.weight]
Loading weights: 3%|▎ | 13/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.mlp.gate_proj.weight]
Loading weights: 3%|▎ | 13/473 [00:00<00:58, 7.93it/s, Materializing param=language_model.layers.0.mlp.gate_proj.weight]
Loading weights: 3%|▎ | 14/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.0.mlp.up_proj.weight]
Loading weights: 3%|▎ | 14/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.0.mlp.up_proj.weight]
Loading weights: 3%|▎ | 15/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.0.post_attention_layernorm.weight]
Loading weights: 3%|▎ | 15/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.0.post_attention_layernorm.weight]
Loading weights: 3%|▎ | 16/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.input_layernorm.weight]
Loading weights: 3%|▎ | 16/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.input_layernorm.weight]
Loading weights: 4%|▎ | 17/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.A_log]
Loading weights: 4%|▎ | 17/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.A_log]
Loading weights: 4%|▍ | 18/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.conv1d.weight]
Loading weights: 4%|▍ | 18/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.conv1d.weight]
Loading weights: 4%|▍ | 19/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.dt_bias]
Loading weights: 4%|▍ | 19/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.dt_bias]
Loading weights: 4%|▍ | 20/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.in_proj_a.weight]
Loading weights: 4%|▍ | 20/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.in_proj_a.weight]
Loading weights: 4%|▍ | 21/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.in_proj_b.weight]
Loading weights: 4%|▍ | 21/473 [00:00<00:57, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.in_proj_b.weight]
Loading weights: 5%|▍ | 22/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.in_proj_qkv.weight]
Loading weights: 5%|▍ | 22/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.in_proj_qkv.weight]
Loading weights: 5%|▍ | 23/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.in_proj_z.weight]
Loading weights: 5%|▍ | 23/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.in_proj_z.weight]
Loading weights: 5%|▌ | 24/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.norm.weight]
Loading weights: 5%|▌ | 24/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.norm.weight]
Loading weights: 5%|▌ | 25/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.out_proj.weight]
Loading weights: 5%|▌ | 25/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.linear_attn.out_proj.weight]
Loading weights: 5%|▌ | 26/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.mlp.down_proj.weight]
Loading weights: 5%|▌ | 26/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.mlp.down_proj.weight]
Loading weights: 6%|▌ | 27/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.mlp.gate_proj.weight]
Loading weights: 6%|▌ | 27/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.mlp.gate_proj.weight]
Loading weights: 6%|▌ | 28/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.mlp.up_proj.weight]
Loading weights: 6%|▌ | 28/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.mlp.up_proj.weight]
Loading weights: 6%|▌ | 29/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.post_attention_layernorm.weight]
Loading weights: 6%|▌ | 29/473 [00:00<00:56, 7.93it/s, Materializing param=language_model.layers.1.post_attention_layernorm.weight]
Loading weights: 6%|▋ | 30/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.input_layernorm.weight]
Loading weights: 6%|▋ | 30/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.input_layernorm.weight]
Loading weights: 7%|▋ | 31/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.A_log]
Loading weights: 7%|▋ | 31/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.A_log]
Loading weights: 7%|▋ | 32/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.conv1d.weight]
Loading weights: 7%|▋ | 32/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.conv1d.weight]
Loading weights: 7%|▋ | 33/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.dt_bias]
Loading weights: 7%|▋ | 33/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.dt_bias]
Loading weights: 7%|▋ | 34/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.in_proj_a.weight]
Loading weights: 7%|▋ | 34/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.in_proj_a.weight]
Loading weights: 7%|▋ | 35/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.in_proj_b.weight]
Loading weights: 7%|▋ | 35/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.in_proj_b.weight]
Loading weights: 8%|▊ | 36/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.in_proj_qkv.weight]
Loading weights: 8%|▊ | 36/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.in_proj_qkv.weight]
Loading weights: 8%|▊ | 37/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.in_proj_z.weight]
Loading weights: 8%|▊ | 37/473 [00:00<00:55, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.in_proj_z.weight]
Loading weights: 8%|▊ | 38/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.norm.weight]
Loading weights: 8%|▊ | 38/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.norm.weight]
Loading weights: 8%|▊ | 39/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.out_proj.weight]
Loading weights: 8%|▊ | 39/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.linear_attn.out_proj.weight]
Loading weights: 8%|▊ | 40/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.mlp.down_proj.weight]
Loading weights: 8%|▊ | 40/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.mlp.down_proj.weight]
Loading weights: 9%|▊ | 41/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.mlp.gate_proj.weight]
Loading weights: 9%|▊ | 41/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.mlp.gate_proj.weight]
Loading weights: 9%|▉ | 42/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.mlp.up_proj.weight]
Loading weights: 9%|▉ | 42/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.mlp.up_proj.weight]
Loading weights: 9%|▉ | 43/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.post_attention_layernorm.weight]
Loading weights: 9%|▉ | 43/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.2.post_attention_layernorm.weight]
Loading weights: 9%|▉ | 44/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.3.input_layernorm.weight]
Loading weights: 9%|▉ | 44/473 [00:00<00:54, 7.93it/s, Materializing param=language_model.layers.3.input_layernorm.weight]
Loading weights: 10%|▉ | 45/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.mlp.down_proj.weight]
Loading weights: 10%|▉ | 45/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.mlp.down_proj.weight]
Loading weights: 10%|▉ | 46/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.mlp.gate_proj.weight]
Loading weights: 10%|▉ | 46/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.mlp.gate_proj.weight]
Loading weights: 10%|▉ | 47/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.mlp.up_proj.weight]
Loading weights: 10%|▉ | 47/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.mlp.up_proj.weight]
Loading weights: 10%|█ | 48/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.post_attention_layernorm.weight]
Loading weights: 10%|█ | 48/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.post_attention_layernorm.weight]
Loading weights: 10%|█ | 49/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.self_attn.k_norm.weight]
Loading weights: 10%|█ | 49/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.self_attn.k_norm.weight]
Loading weights: 11%|█ | 50/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.self_attn.k_proj.weight]
Loading weights: 11%|█ | 50/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.self_attn.k_proj.weight]
Loading weights: 11%|█ | 51/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.self_attn.o_proj.weight]
Loading weights: 11%|█ | 51/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.self_attn.o_proj.weight]
Loading weights: 11%|█ | 52/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.self_attn.q_norm.weight]
Loading weights: 11%|█ | 52/473 [00:00<00:53, 7.93it/s, Materializing param=language_model.layers.3.self_attn.q_norm.weight]
Loading weights: 11%|█ | 53/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.3.self_attn.q_proj.weight]
Loading weights: 11%|█ | 53/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.3.self_attn.q_proj.weight]
Loading weights: 11%|█▏ | 54/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.3.self_attn.v_proj.weight]
Loading weights: 11%|█▏ | 54/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.3.self_attn.v_proj.weight]
Loading weights: 12%|█▏ | 55/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.input_layernorm.weight]
Loading weights: 12%|█▏ | 55/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.input_layernorm.weight]
Loading weights: 12%|█▏ | 56/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.A_log]
Loading weights: 12%|█▏ | 56/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.A_log]
Loading weights: 12%|█▏ | 57/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.conv1d.weight]
Loading weights: 12%|█▏ | 57/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.conv1d.weight]
Loading weights: 12%|█▏ | 58/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.dt_bias]
Loading weights: 12%|█▏ | 58/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.dt_bias]
Loading weights: 12%|█▏ | 59/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.in_proj_a.weight]
Loading weights: 12%|█▏ | 59/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.in_proj_a.weight]
Loading weights: 13%|█▎ | 60/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.in_proj_b.weight]
Loading weights: 13%|█▎ | 60/473 [00:00<00:52, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.in_proj_b.weight]
Loading weights: 13%|█▎ | 61/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.in_proj_qkv.weight]
Loading weights: 13%|█▎ | 61/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.in_proj_qkv.weight]
Loading weights: 13%|█▎ | 62/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.in_proj_z.weight]
Loading weights: 13%|█▎ | 62/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.in_proj_z.weight]
Loading weights: 13%|█▎ | 63/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.norm.weight]
Loading weights: 13%|█▎ | 63/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.norm.weight]
Loading weights: 14%|█▎ | 64/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.out_proj.weight]
Loading weights: 14%|█▎ | 64/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.linear_attn.out_proj.weight]
Loading weights: 14%|█▎ | 65/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.mlp.down_proj.weight]
Loading weights: 14%|█▎ | 65/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.mlp.down_proj.weight]
Loading weights: 14%|█▍ | 66/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.mlp.gate_proj.weight]
Loading weights: 14%|█▍ | 66/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.mlp.gate_proj.weight]
Loading weights: 14%|█▍ | 67/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.mlp.up_proj.weight]
Loading weights: 14%|█▍ | 67/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.mlp.up_proj.weight]
Loading weights: 14%|█▍ | 68/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.post_attention_layernorm.weight]
Loading weights: 14%|█▍ | 68/473 [00:00<00:51, 7.93it/s, Materializing param=language_model.layers.4.post_attention_layernorm.weight]
Loading weights: 15%|█▍ | 69/473 [00:00<00:50, 7.93it/s, Materializing param=language_model.layers.5.input_layernorm.weight]
Loading weights: 15%|█▍ | 69/473 [00:00<00:50, 7.93it/s, Materializing param=language_model.layers.5.input_layernorm.weight]
Loading weights: 15%|█▍ | 70/473 [00:00<00:50, 7.93it/s, Materializing param=language_model.layers.5.linear_attn.A_log]
Loading weights: 15%|█▍ | 70/473 [00:00<00:50, 7.93it/s, Materializing param=language_model.layers.5.linear_attn.A_log]
Loading weights: 15%|█▌ | 71/473 [00:00<00:50, 7.93it/s, Materializing param=language_model.layers.5.linear_attn.conv1d.weight]
Loading weights: 15%|█▌ | 71/473 [00:00<00:50, 7.93it/s, Materializing param=language_model.layers.5.linear_attn.conv1d.weight]
Loading weights: 15%|█▌ | 72/473 [00:00<00:50, 7.93it/s, Materializing param=language_model.layers.5.linear_attn.dt_bias]
Loading weights: 15%|█▌ | 72/473 [00:00<00:50, 7.93it/s, Materializing param=language_model.layers.5.linear_attn.dt_bias]
Loading weights: 15%|█▌ | 73/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.dt_bias]
Loading weights: 15%|█▌ | 73/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.in_proj_a.weight]
Loading weights: 15%|█▌ | 73/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.in_proj_a.weight]
Loading weights: 16%|█▌ | 74/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.in_proj_b.weight]
Loading weights: 16%|█▌ | 74/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.in_proj_b.weight]
Loading weights: 16%|█▌ | 75/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.in_proj_qkv.weight]
Loading weights: 16%|█▌ | 75/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.in_proj_qkv.weight]
Loading weights: 16%|█▌ | 76/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.in_proj_z.weight]
Loading weights: 16%|█▌ | 76/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.in_proj_z.weight]
Loading weights: 16%|█▋ | 77/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.norm.weight]
Loading weights: 16%|█▋ | 77/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.norm.weight]
Loading weights: 16%|█▋ | 78/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.out_proj.weight]
Loading weights: 16%|█▋ | 78/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.linear_attn.out_proj.weight]
Loading weights: 17%|█▋ | 79/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.mlp.down_proj.weight]
Loading weights: 17%|█▋ | 79/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.mlp.down_proj.weight]
Loading weights: 17%|█▋ | 80/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.mlp.gate_proj.weight]
Loading weights: 17%|█▋ | 80/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.mlp.gate_proj.weight]
Loading weights: 17%|█▋ | 81/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.mlp.up_proj.weight]
Loading weights: 17%|█▋ | 81/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.mlp.up_proj.weight]
Loading weights: 17%|█▋ | 82/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.post_attention_layernorm.weight]
Loading weights: 17%|█▋ | 82/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.5.post_attention_layernorm.weight]
Loading weights: 18%|█▊ | 83/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.input_layernorm.weight]
Loading weights: 18%|█▊ | 83/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.input_layernorm.weight]
Loading weights: 18%|█▊ | 84/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.A_log]
Loading weights: 18%|█▊ | 84/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.A_log]
Loading weights: 18%|█▊ | 85/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.conv1d.weight]
Loading weights: 18%|█▊ | 85/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.conv1d.weight]
Loading weights: 18%|█▊ | 86/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.dt_bias]
Loading weights: 18%|█▊ | 86/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.dt_bias]
Loading weights: 18%|█▊ | 87/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.in_proj_a.weight]
Loading weights: 18%|█▊ | 87/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.in_proj_a.weight]
Loading weights: 19%|█▊ | 88/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.in_proj_b.weight]
Loading weights: 19%|█▊ | 88/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.in_proj_b.weight]
Loading weights: 19%|█▉ | 89/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.in_proj_qkv.weight]
Loading weights: 19%|█▉ | 89/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.in_proj_qkv.weight]
Loading weights: 19%|█▉ | 90/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.in_proj_z.weight]
Loading weights: 19%|█▉ | 90/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.in_proj_z.weight]
Loading weights: 19%|█▉ | 91/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.norm.weight]
Loading weights: 19%|█▉ | 91/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.norm.weight]
Loading weights: 19%|█▉ | 92/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.out_proj.weight]
Loading weights: 19%|█▉ | 92/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.linear_attn.out_proj.weight]
Loading weights: 20%|█▉ | 93/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.mlp.down_proj.weight]
Loading weights: 20%|█▉ | 93/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.mlp.down_proj.weight]
Loading weights: 20%|█▉ | 94/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.mlp.gate_proj.weight]
Loading weights: 20%|█▉ | 94/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.mlp.gate_proj.weight]
Loading weights: 20%|██ | 95/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.mlp.up_proj.weight]
Loading weights: 20%|██ | 95/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.mlp.up_proj.weight]
Loading weights: 20%|██ | 96/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.post_attention_layernorm.weight]
Loading weights: 20%|██ | 96/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.6.post_attention_layernorm.weight]
Loading weights: 21%|██ | 97/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.input_layernorm.weight]
Loading weights: 21%|██ | 97/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.input_layernorm.weight]
Loading weights: 21%|██ | 98/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.mlp.down_proj.weight]
Loading weights: 21%|██ | 98/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.mlp.down_proj.weight]
Loading weights: 21%|██ | 99/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.mlp.gate_proj.weight]
Loading weights: 21%|██ | 99/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.mlp.gate_proj.weight]
Loading weights: 21%|██ | 100/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.mlp.up_proj.weight]
Loading weights: 21%|██ | 100/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.mlp.up_proj.weight]
Loading weights: 21%|██▏ | 101/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.post_attention_layernorm.weight]
Loading weights: 21%|██▏ | 101/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.post_attention_layernorm.weight]
Loading weights: 22%|██▏ | 102/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.k_norm.weight]
Loading weights: 22%|██▏ | 102/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.k_norm.weight]
Loading weights: 22%|██▏ | 103/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.k_proj.weight]
Loading weights: 22%|██▏ | 103/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.k_proj.weight]
Loading weights: 22%|██▏ | 104/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.o_proj.weight]
Loading weights: 22%|██▏ | 104/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.o_proj.weight]
Loading weights: 22%|██▏ | 105/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.q_norm.weight]
Loading weights: 22%|██▏ | 105/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.q_norm.weight]
Loading weights: 22%|██▏ | 106/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.q_proj.weight]
Loading weights: 22%|██▏ | 106/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.q_proj.weight]
Loading weights: 23%|██▎ | 107/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.v_proj.weight]
Loading weights: 23%|██▎ | 107/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.7.self_attn.v_proj.weight]
Loading weights: 23%|██▎ | 108/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.input_layernorm.weight]
Loading weights: 23%|██▎ | 108/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.input_layernorm.weight]
Loading weights: 23%|██▎ | 109/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.A_log]
Loading weights: 23%|██▎ | 109/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.A_log]
Loading weights: 23%|██▎ | 110/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.conv1d.weight]
Loading weights: 23%|██▎ | 110/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.conv1d.weight]
Loading weights: 23%|██▎ | 111/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.dt_bias]
Loading weights: 23%|██▎ | 111/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.dt_bias]
Loading weights: 24%|██▎ | 112/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_a.weight]
Loading weights: 24%|██▎ | 112/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_a.weight]
Loading weights: 24%|██▍ | 113/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_b.weight]
Loading weights: 24%|██▍ | 113/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_b.weight]
Loading weights: 24%|██▍ | 114/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_qkv.weight]
Loading weights: 24%|██▍ | 114/473 [00:00<00:01, 260.07it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_qkv.weight]
Loading weights: 24%|██▍ | 115/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_qkv.weight]
Loading weights: 24%|██▍ | 115/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_z.weight]
Loading weights: 24%|██▍ | 115/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.linear_attn.in_proj_z.weight]
Loading weights: 25%|██▍ | 116/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.linear_attn.norm.weight]
Loading weights: 25%|██▍ | 116/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.linear_attn.norm.weight]
Loading weights: 25%|██▍ | 117/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.linear_attn.out_proj.weight]
Loading weights: 25%|██▍ | 117/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.linear_attn.out_proj.weight]
Loading weights: 25%|██▍ | 118/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.mlp.down_proj.weight]
Loading weights: 25%|██▍ | 118/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.mlp.down_proj.weight]
Loading weights: 25%|██▌ | 119/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.mlp.gate_proj.weight]
Loading weights: 25%|██▌ | 119/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.mlp.gate_proj.weight]
Loading weights: 25%|██▌ | 120/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.mlp.up_proj.weight]
Loading weights: 25%|██▌ | 120/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.mlp.up_proj.weight]
Loading weights: 26%|██▌ | 121/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.post_attention_layernorm.weight]
Loading weights: 26%|██▌ | 121/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.8.post_attention_layernorm.weight]
Loading weights: 26%|██▌ | 122/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.input_layernorm.weight]
Loading weights: 26%|██▌ | 122/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.input_layernorm.weight]
Loading weights: 26%|██▌ | 123/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.A_log]
Loading weights: 26%|██▌ | 123/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.A_log]
Loading weights: 26%|██▌ | 124/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.conv1d.weight]
Loading weights: 26%|██▌ | 124/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.conv1d.weight]
Loading weights: 26%|██▋ | 125/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.dt_bias]
Loading weights: 26%|██▋ | 125/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.dt_bias]
Loading weights: 27%|██▋ | 126/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.in_proj_a.weight]
Loading weights: 27%|██▋ | 126/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.in_proj_a.weight]
Loading weights: 27%|██▋ | 127/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.in_proj_b.weight]
Loading weights: 27%|██▋ | 127/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.in_proj_b.weight]
Loading weights: 27%|██▋ | 128/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.in_proj_qkv.weight]
Loading weights: 27%|██▋ | 128/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.in_proj_qkv.weight]
Loading weights: 27%|██▋ | 129/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.in_proj_z.weight]
Loading weights: 27%|██▋ | 129/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.in_proj_z.weight]
Loading weights: 27%|██▋ | 130/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.norm.weight]
Loading weights: 27%|██▋ | 130/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.norm.weight]
Loading weights: 28%|██▊ | 131/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.out_proj.weight]
Loading weights: 28%|██▊ | 131/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.linear_attn.out_proj.weight]
Loading weights: 28%|██▊ | 132/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.mlp.down_proj.weight]
Loading weights: 28%|██▊ | 132/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.mlp.down_proj.weight]
Loading weights: 28%|██▊ | 133/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.mlp.gate_proj.weight]
Loading weights: 28%|██▊ | 133/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.mlp.gate_proj.weight]
Loading weights: 28%|██▊ | 134/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.mlp.up_proj.weight]
Loading weights: 28%|██▊ | 134/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.mlp.up_proj.weight]
Loading weights: 29%|██▊ | 135/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.post_attention_layernorm.weight]
Loading weights: 29%|██▊ | 135/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.9.post_attention_layernorm.weight]
Loading weights: 29%|██▉ | 136/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.input_layernorm.weight]
Loading weights: 29%|██▉ | 136/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.input_layernorm.weight]
Loading weights: 29%|██▉ | 137/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.A_log]
Loading weights: 29%|██▉ | 137/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.A_log]
Loading weights: 29%|██▉ | 138/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.conv1d.weight]
Loading weights: 29%|██▉ | 138/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.conv1d.weight]
Loading weights: 29%|██▉ | 139/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.dt_bias]
Loading weights: 29%|██▉ | 139/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.dt_bias]
Loading weights: 30%|██▉ | 140/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.in_proj_a.weight]
Loading weights: 30%|██▉ | 140/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.in_proj_a.weight]
Loading weights: 30%|██▉ | 141/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.in_proj_b.weight]
Loading weights: 30%|██▉ | 141/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.in_proj_b.weight]
Loading weights: 30%|███ | 142/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.in_proj_qkv.weight]
Loading weights: 30%|███ | 142/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.in_proj_qkv.weight]
Loading weights: 30%|███ | 143/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.in_proj_z.weight]
Loading weights: 30%|███ | 143/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.in_proj_z.weight]
Loading weights: 30%|███ | 144/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.norm.weight]
Loading weights: 30%|███ | 144/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.norm.weight]
Loading weights: 31%|███ | 145/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.out_proj.weight]
Loading weights: 31%|███ | 145/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.linear_attn.out_proj.weight]
Loading weights: 31%|███ | 146/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.mlp.down_proj.weight]
Loading weights: 31%|███ | 146/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.mlp.down_proj.weight]
Loading weights: 31%|███ | 147/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.mlp.gate_proj.weight]
Loading weights: 31%|███ | 147/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.mlp.gate_proj.weight]
Loading weights: 31%|███▏ | 148/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.mlp.up_proj.weight]
Loading weights: 31%|███▏ | 148/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.mlp.up_proj.weight]
Loading weights: 32%|███▏ | 149/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.post_attention_layernorm.weight]
Loading weights: 32%|███▏ | 149/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.10.post_attention_layernorm.weight]
Loading weights: 32%|███▏ | 150/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.input_layernorm.weight]
Loading weights: 32%|███▏ | 150/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.input_layernorm.weight]
Loading weights: 32%|███▏ | 151/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.mlp.down_proj.weight]
Loading weights: 32%|███▏ | 151/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.mlp.down_proj.weight]
Loading weights: 32%|███▏ | 152/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.mlp.gate_proj.weight]
Loading weights: 32%|███▏ | 152/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.mlp.gate_proj.weight]
Loading weights: 32%|███▏ | 153/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.mlp.up_proj.weight]
Loading weights: 32%|███▏ | 153/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.mlp.up_proj.weight]
Loading weights: 33%|███▎ | 154/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.post_attention_layernorm.weight]
Loading weights: 33%|███▎ | 154/473 [00:00<00:01, 287.24it/s, Materializing param=language_model.layers.11.post_attention_layernorm.weight]
Loading weights: 33%|███▎ | 155/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.post_attention_layernorm.weight]
Loading weights: 33%|███▎ | 155/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.k_norm.weight]
Loading weights: 33%|███▎ | 155/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.k_norm.weight]
Loading weights: 33%|███▎ | 156/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.k_proj.weight]
Loading weights: 33%|███▎ | 156/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.k_proj.weight]
Loading weights: 33%|███▎ | 157/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.o_proj.weight]
Loading weights: 33%|███▎ | 157/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.o_proj.weight]
Loading weights: 33%|███▎ | 158/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.q_norm.weight]
Loading weights: 33%|███▎ | 158/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.q_norm.weight]
Loading weights: 34%|███▎ | 159/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.q_proj.weight]
Loading weights: 34%|███▎ | 159/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.q_proj.weight]
Loading weights: 34%|███▍ | 160/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.v_proj.weight]
Loading weights: 34%|███▍ | 160/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.11.self_attn.v_proj.weight]
Loading weights: 34%|███▍ | 161/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.input_layernorm.weight]
Loading weights: 34%|███▍ | 161/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.input_layernorm.weight]
Loading weights: 34%|███▍ | 162/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.A_log]
Loading weights: 34%|███▍ | 162/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.A_log]
Loading weights: 34%|███▍ | 163/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.conv1d.weight]
Loading weights: 34%|███▍ | 163/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.conv1d.weight]
Loading weights: 35%|███▍ | 164/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.dt_bias]
Loading weights: 35%|███▍ | 164/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.dt_bias]
Loading weights: 35%|███▍ | 165/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.in_proj_a.weight]
Loading weights: 35%|███▍ | 165/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.in_proj_a.weight]
Loading weights: 35%|███▌ | 166/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.in_proj_b.weight]
Loading weights: 35%|███▌ | 166/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.in_proj_b.weight]
Loading weights: 35%|███▌ | 167/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.in_proj_qkv.weight]
Loading weights: 35%|███▌ | 167/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.in_proj_qkv.weight]
Loading weights: 36%|███▌ | 168/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.in_proj_z.weight]
Loading weights: 36%|███▌ | 168/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.in_proj_z.weight]
Loading weights: 36%|███▌ | 169/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.norm.weight]
Loading weights: 36%|███▌ | 169/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.norm.weight]
Loading weights: 36%|███▌ | 170/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.out_proj.weight]
Loading weights: 36%|███▌ | 170/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.linear_attn.out_proj.weight]
Loading weights: 36%|███▌ | 171/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.mlp.down_proj.weight]
Loading weights: 36%|███▌ | 171/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.mlp.down_proj.weight]
Loading weights: 36%|███▋ | 172/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.mlp.gate_proj.weight]
Loading weights: 36%|███▋ | 172/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.mlp.gate_proj.weight]
Loading weights: 37%|███▋ | 173/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.mlp.up_proj.weight]
Loading weights: 37%|███▋ | 173/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.mlp.up_proj.weight]
Loading weights: 37%|███▋ | 174/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.post_attention_layernorm.weight]
Loading weights: 37%|███▋ | 174/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.12.post_attention_layernorm.weight]
Loading weights: 37%|███▋ | 175/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.input_layernorm.weight]
Loading weights: 37%|███▋ | 175/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.input_layernorm.weight]
Loading weights: 37%|███▋ | 176/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.A_log]
Loading weights: 37%|███▋ | 176/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.A_log]
Loading weights: 37%|███▋ | 177/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.conv1d.weight]
Loading weights: 37%|███▋ | 177/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.conv1d.weight]
Loading weights: 38%|███▊ | 178/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.dt_bias]
Loading weights: 38%|███▊ | 178/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.dt_bias]
Loading weights: 38%|███▊ | 179/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.in_proj_a.weight]
Loading weights: 38%|███▊ | 179/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.in_proj_a.weight]
Loading weights: 38%|███▊ | 180/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.in_proj_b.weight]
Loading weights: 38%|███▊ | 180/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.in_proj_b.weight]
Loading weights: 38%|███▊ | 181/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.in_proj_qkv.weight]
Loading weights: 38%|███▊ | 181/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.in_proj_qkv.weight]
Loading weights: 38%|███▊ | 182/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.in_proj_z.weight]
Loading weights: 38%|███▊ | 182/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.in_proj_z.weight]
Loading weights: 39%|███▊ | 183/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.norm.weight]
Loading weights: 39%|███▊ | 183/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.norm.weight]
Loading weights: 39%|███▉ | 184/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.out_proj.weight]
Loading weights: 39%|███▉ | 184/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.linear_attn.out_proj.weight]
Loading weights: 39%|███▉ | 185/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.mlp.down_proj.weight]
Loading weights: 39%|███▉ | 185/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.mlp.down_proj.weight]
Loading weights: 39%|███▉ | 186/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.mlp.gate_proj.weight]
Loading weights: 39%|███▉ | 186/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.mlp.gate_proj.weight]
Loading weights: 40%|███▉ | 187/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.mlp.up_proj.weight]
Loading weights: 40%|███▉ | 187/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.mlp.up_proj.weight]
Loading weights: 40%|███▉ | 188/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.post_attention_layernorm.weight]
Loading weights: 40%|███▉ | 188/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.13.post_attention_layernorm.weight]
Loading weights: 40%|███▉ | 189/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.input_layernorm.weight]
Loading weights: 40%|███▉ | 189/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.input_layernorm.weight]
Loading weights: 40%|████ | 190/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.A_log]
Loading weights: 40%|████ | 190/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.A_log]
Loading weights: 40%|████ | 191/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.conv1d.weight]
Loading weights: 40%|████ | 191/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.conv1d.weight]
Loading weights: 41%|████ | 192/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.dt_bias]
Loading weights: 41%|████ | 192/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.dt_bias]
Loading weights: 41%|████ | 193/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.in_proj_a.weight]
Loading weights: 41%|████ | 193/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.in_proj_a.weight]
Loading weights: 41%|████ | 194/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.in_proj_b.weight]
Loading weights: 41%|████ | 194/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.in_proj_b.weight]
Loading weights: 41%|████ | 195/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.in_proj_qkv.weight]
Loading weights: 41%|████ | 195/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.in_proj_qkv.weight]
Loading weights: 41%|████▏ | 196/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.in_proj_z.weight]
Loading weights: 41%|████▏ | 196/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.in_proj_z.weight]
Loading weights: 42%|████▏ | 197/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.norm.weight]
Loading weights: 42%|████▏ | 197/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.norm.weight]
Loading weights: 42%|████▏ | 198/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.out_proj.weight]
Loading weights: 42%|████▏ | 198/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.linear_attn.out_proj.weight]
Loading weights: 42%|████▏ | 199/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.mlp.down_proj.weight]
Loading weights: 42%|████▏ | 199/473 [00:00<00:00, 321.81it/s, Materializing param=language_model.layers.14.mlp.down_proj.weight]
Loading weights: 42%|████▏ | 200/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.14.mlp.down_proj.weight]
Loading weights: 42%|████▏ | 200/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.14.mlp.gate_proj.weight]
Loading weights: 42%|████▏ | 200/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.14.mlp.gate_proj.weight]
Loading weights: 42%|████▏ | 201/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.14.mlp.up_proj.weight]
Loading weights: 42%|████▏ | 201/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.14.mlp.up_proj.weight]
Loading weights: 43%|████▎ | 202/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.14.post_attention_layernorm.weight]
Loading weights: 43%|████▎ | 202/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.14.post_attention_layernorm.weight]
Loading weights: 43%|████▎ | 203/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.input_layernorm.weight]
Loading weights: 43%|████▎ | 203/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.input_layernorm.weight]
Loading weights: 43%|████▎ | 204/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.mlp.down_proj.weight]
Loading weights: 43%|████▎ | 204/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.mlp.down_proj.weight]
Loading weights: 43%|████▎ | 205/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.mlp.gate_proj.weight]
Loading weights: 43%|████▎ | 205/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.mlp.gate_proj.weight]
Loading weights: 44%|████▎ | 206/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.mlp.up_proj.weight]
Loading weights: 44%|████▎ | 206/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.mlp.up_proj.weight]
Loading weights: 44%|████▍ | 207/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.post_attention_layernorm.weight]
Loading weights: 44%|████▍ | 207/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.post_attention_layernorm.weight]
Loading weights: 44%|████▍ | 208/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.k_norm.weight]
Loading weights: 44%|████▍ | 208/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.k_norm.weight]
Loading weights: 44%|████▍ | 209/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.k_proj.weight]
Loading weights: 44%|████▍ | 209/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.k_proj.weight]
Loading weights: 44%|████▍ | 210/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.o_proj.weight]
Loading weights: 44%|████▍ | 210/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.o_proj.weight]
Loading weights: 45%|████▍ | 211/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.q_norm.weight]
Loading weights: 45%|████▍ | 211/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.q_norm.weight]
Loading weights: 45%|████▍ | 212/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.q_proj.weight]
Loading weights: 45%|████▍ | 212/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.q_proj.weight]
Loading weights: 45%|████▌ | 213/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.v_proj.weight]
Loading weights: 45%|████▌ | 213/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.15.self_attn.v_proj.weight]
Loading weights: 45%|████▌ | 214/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.input_layernorm.weight]
Loading weights: 45%|████▌ | 214/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.input_layernorm.weight]
Loading weights: 45%|████▌ | 215/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.A_log]
Loading weights: 45%|████▌ | 215/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.A_log]
Loading weights: 46%|████▌ | 216/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.conv1d.weight]
Loading weights: 46%|████▌ | 216/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.conv1d.weight]
Loading weights: 46%|████▌ | 217/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.dt_bias]
Loading weights: 46%|████▌ | 217/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.dt_bias]
Loading weights: 46%|████▌ | 218/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.in_proj_a.weight]
Loading weights: 46%|████▌ | 218/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.in_proj_a.weight]
Loading weights: 46%|████▋ | 219/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.in_proj_b.weight]
Loading weights: 46%|████▋ | 219/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.in_proj_b.weight]
Loading weights: 47%|████▋ | 220/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.in_proj_qkv.weight]
Loading weights: 47%|████▋ | 220/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.in_proj_qkv.weight]
Loading weights: 47%|████▋ | 221/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.in_proj_z.weight]
Loading weights: 47%|████▋ | 221/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.in_proj_z.weight]
Loading weights: 47%|████▋ | 222/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.norm.weight]
Loading weights: 47%|████▋ | 222/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.norm.weight]
Loading weights: 47%|████▋ | 223/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.out_proj.weight]
Loading weights: 47%|████▋ | 223/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.linear_attn.out_proj.weight]
Loading weights: 47%|████▋ | 224/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.mlp.down_proj.weight]
Loading weights: 47%|████▋ | 224/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.mlp.down_proj.weight]
Loading weights: 48%|████▊ | 225/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.mlp.gate_proj.weight]
Loading weights: 48%|████▊ | 225/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.mlp.gate_proj.weight]
Loading weights: 48%|████▊ | 226/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.mlp.up_proj.weight]
Loading weights: 48%|████▊ | 226/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.mlp.up_proj.weight]
Loading weights: 48%|████▊ | 227/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.post_attention_layernorm.weight]
Loading weights: 48%|████▊ | 227/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.16.post_attention_layernorm.weight]
Loading weights: 48%|████▊ | 228/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.input_layernorm.weight]
Loading weights: 48%|████▊ | 228/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.input_layernorm.weight]
Loading weights: 48%|████▊ | 229/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.A_log]
Loading weights: 48%|████▊ | 229/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.A_log]
Loading weights: 49%|████▊ | 230/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.conv1d.weight]
Loading weights: 49%|████▊ | 230/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.conv1d.weight]
Loading weights: 49%|████▉ | 231/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.dt_bias]
Loading weights: 49%|████▉ | 231/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.dt_bias]
Loading weights: 49%|████▉ | 232/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.in_proj_a.weight]
Loading weights: 49%|████▉ | 232/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.in_proj_a.weight]
Loading weights: 49%|████▉ | 233/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.in_proj_b.weight]
Loading weights: 49%|████▉ | 233/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.in_proj_b.weight]
Loading weights: 49%|████▉ | 234/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.in_proj_qkv.weight]
Loading weights: 49%|████▉ | 234/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.in_proj_qkv.weight]
Loading weights: 50%|████▉ | 235/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.in_proj_z.weight]
Loading weights: 50%|████▉ | 235/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.in_proj_z.weight]
Loading weights: 50%|████▉ | 236/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.norm.weight]
Loading weights: 50%|████▉ | 236/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.norm.weight]
Loading weights: 50%|█████ | 237/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.out_proj.weight]
Loading weights: 50%|█████ | 237/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.linear_attn.out_proj.weight]
Loading weights: 50%|█████ | 238/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.mlp.down_proj.weight]
Loading weights: 50%|█████ | 238/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.mlp.down_proj.weight]
Loading weights: 51%|█████ | 239/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.mlp.gate_proj.weight]
Loading weights: 51%|█████ | 239/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.mlp.gate_proj.weight]
Loading weights: 51%|█████ | 240/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.mlp.up_proj.weight]
Loading weights: 51%|█████ | 240/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.mlp.up_proj.weight]
Loading weights: 51%|█████ | 241/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.post_attention_layernorm.weight]
Loading weights: 51%|█████ | 241/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.17.post_attention_layernorm.weight]
Loading weights: 51%|█████ | 242/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.input_layernorm.weight]
Loading weights: 51%|█████ | 242/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.input_layernorm.weight]
Loading weights: 51%|█████▏ | 243/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.A_log]
Loading weights: 51%|█████▏ | 243/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.A_log]
Loading weights: 52%|█████▏ | 244/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.conv1d.weight]
Loading weights: 52%|█████▏ | 244/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.conv1d.weight]
Loading weights: 52%|█████▏ | 245/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.dt_bias]
Loading weights: 52%|█████▏ | 245/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.dt_bias]
Loading weights: 52%|█████▏ | 246/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.in_proj_a.weight]
Loading weights: 52%|█████▏ | 246/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.in_proj_a.weight]
Loading weights: 52%|█████▏ | 247/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.in_proj_b.weight]
Loading weights: 52%|█████▏ | 247/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.in_proj_b.weight]
Loading weights: 52%|█████▏ | 248/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.in_proj_qkv.weight]
Loading weights: 52%|█████▏ | 248/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.in_proj_qkv.weight]
Loading weights: 53%|█████▎ | 249/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.in_proj_z.weight]
Loading weights: 53%|█████▎ | 249/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.in_proj_z.weight]
Loading weights: 53%|█████▎ | 250/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.norm.weight]
Loading weights: 53%|█████▎ | 250/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.norm.weight]
Loading weights: 53%|█████▎ | 251/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.out_proj.weight]
Loading weights: 53%|█████▎ | 251/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.linear_attn.out_proj.weight]
Loading weights: 53%|█████▎ | 252/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.mlp.down_proj.weight]
Loading weights: 53%|█████▎ | 252/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.mlp.down_proj.weight]
Loading weights: 53%|█████▎ | 253/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.mlp.gate_proj.weight]
Loading weights: 53%|█████▎ | 253/473 [00:00<00:00, 359.68it/s, Materializing param=language_model.layers.18.mlp.gate_proj.weight]
Loading weights: 54%|█████▎ | 254/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.18.mlp.gate_proj.weight]
Loading weights: 54%|█████▎ | 254/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.18.mlp.up_proj.weight]
Loading weights: 54%|█████▎ | 254/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.18.mlp.up_proj.weight]
Loading weights: 54%|█████▍ | 255/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.18.post_attention_layernorm.weight]
Loading weights: 54%|█████▍ | 255/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.18.post_attention_layernorm.weight]
Loading weights: 54%|█████▍ | 256/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.input_layernorm.weight]
Loading weights: 54%|█████▍ | 256/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.input_layernorm.weight]
Loading weights: 54%|█████▍ | 257/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.mlp.down_proj.weight]
Loading weights: 54%|█████▍ | 257/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.mlp.down_proj.weight]
Loading weights: 55%|█████▍ | 258/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.mlp.gate_proj.weight]
Loading weights: 55%|█████▍ | 258/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.mlp.gate_proj.weight]
Loading weights: 55%|█████▍ | 259/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.mlp.up_proj.weight]
Loading weights: 55%|█████▍ | 259/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.mlp.up_proj.weight]
Loading weights: 55%|█████▍ | 260/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.post_attention_layernorm.weight]
Loading weights: 55%|█████▍ | 260/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.post_attention_layernorm.weight]
Loading weights: 55%|█████▌ | 261/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.k_norm.weight]
Loading weights: 55%|█████▌ | 261/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.k_norm.weight]
Loading weights: 55%|█████▌ | 262/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.k_proj.weight]
Loading weights: 55%|█████▌ | 262/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.k_proj.weight]
Loading weights: 56%|█████▌ | 263/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.o_proj.weight]
Loading weights: 56%|█████▌ | 263/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.o_proj.weight]
Loading weights: 56%|█████▌ | 264/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.q_norm.weight]
Loading weights: 56%|█████▌ | 264/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.q_norm.weight]
Loading weights: 56%|█████▌ | 265/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.q_proj.weight]
Loading weights: 56%|█████▌ | 265/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.q_proj.weight]
Loading weights: 56%|█████▌ | 266/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.v_proj.weight]
Loading weights: 56%|█████▌ | 266/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.19.self_attn.v_proj.weight]
Loading weights: 56%|█████▋ | 267/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.input_layernorm.weight]
Loading weights: 56%|█████▋ | 267/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.input_layernorm.weight]
Loading weights: 57%|█████▋ | 268/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.A_log]
Loading weights: 57%|█████▋ | 268/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.A_log]
Loading weights: 57%|█████▋ | 269/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.conv1d.weight]
Loading weights: 57%|█████▋ | 269/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.conv1d.weight]
Loading weights: 57%|█████▋ | 270/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.dt_bias]
Loading weights: 57%|█████▋ | 270/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.dt_bias]
Loading weights: 57%|█████▋ | 271/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.in_proj_a.weight]
Loading weights: 57%|█████▋ | 271/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.in_proj_a.weight]
Loading weights: 58%|█████▊ | 272/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.in_proj_b.weight]
Loading weights: 58%|█████▊ | 272/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.in_proj_b.weight]
Loading weights: 58%|█████▊ | 273/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.in_proj_qkv.weight]
Loading weights: 58%|█████▊ | 273/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.in_proj_qkv.weight]
Loading weights: 58%|█████▊ | 274/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.in_proj_z.weight]
Loading weights: 58%|█████▊ | 274/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.in_proj_z.weight]
Loading weights: 58%|█████▊ | 275/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.norm.weight]
Loading weights: 58%|█████▊ | 275/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.norm.weight]
Loading weights: 58%|█████▊ | 276/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.out_proj.weight]
Loading weights: 58%|█████▊ | 276/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.linear_attn.out_proj.weight]
Loading weights: 59%|█████▊ | 277/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.mlp.down_proj.weight]
Loading weights: 59%|█████▊ | 277/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.mlp.down_proj.weight]
Loading weights: 59%|█████▉ | 278/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.mlp.gate_proj.weight]
Loading weights: 59%|█████▉ | 278/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.mlp.gate_proj.weight]
Loading weights: 59%|█████▉ | 279/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.mlp.up_proj.weight]
Loading weights: 59%|█████▉ | 279/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.mlp.up_proj.weight]
Loading weights: 59%|█████▉ | 280/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.post_attention_layernorm.weight]
Loading weights: 59%|█████▉ | 280/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.20.post_attention_layernorm.weight]
Loading weights: 59%|█████▉ | 281/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.input_layernorm.weight]
Loading weights: 59%|█████▉ | 281/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.input_layernorm.weight]
Loading weights: 60%|█████▉ | 282/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.A_log]
Loading weights: 60%|█████▉ | 282/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.A_log]
Loading weights: 60%|█████▉ | 283/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.conv1d.weight]
Loading weights: 60%|█████▉ | 283/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.conv1d.weight]
Loading weights: 60%|██████ | 284/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.dt_bias]
Loading weights: 60%|██████ | 284/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.dt_bias]
Loading weights: 60%|██████ | 285/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.in_proj_a.weight]
Loading weights: 60%|██████ | 285/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.in_proj_a.weight]
Loading weights: 60%|██████ | 286/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.in_proj_b.weight]
Loading weights: 60%|██████ | 286/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.in_proj_b.weight]
Loading weights: 61%|██████ | 287/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.in_proj_qkv.weight]
Loading weights: 61%|██████ | 287/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.in_proj_qkv.weight]
Loading weights: 61%|██████ | 288/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.in_proj_z.weight]
Loading weights: 61%|██████ | 288/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.in_proj_z.weight]
Loading weights: 61%|██████ | 289/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.norm.weight]
Loading weights: 61%|██████ | 289/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.norm.weight]
Loading weights: 61%|██████▏ | 290/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.out_proj.weight]
Loading weights: 61%|██████▏ | 290/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.linear_attn.out_proj.weight]
Loading weights: 62%|██████▏ | 291/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.mlp.down_proj.weight]
Loading weights: 62%|██████▏ | 291/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.mlp.down_proj.weight]
Loading weights: 62%|██████▏ | 292/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.mlp.gate_proj.weight]
Loading weights: 62%|██████▏ | 292/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.mlp.gate_proj.weight]
Loading weights: 62%|██████▏ | 293/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.mlp.up_proj.weight]
Loading weights: 62%|██████▏ | 293/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.mlp.up_proj.weight]
Loading weights: 62%|██████▏ | 294/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.post_attention_layernorm.weight]
Loading weights: 62%|██████▏ | 294/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.21.post_attention_layernorm.weight]
Loading weights: 62%|██████▏ | 295/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.22.input_layernorm.weight]
Loading weights: 62%|██████▏ | 295/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.22.input_layernorm.weight]
Loading weights: 63%|██████▎ | 296/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.22.linear_attn.A_log]
Loading weights: 63%|██████▎ | 296/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.22.linear_attn.A_log]
Loading weights: 63%|██████▎ | 297/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.22.linear_attn.conv1d.weight]
Loading weights: 63%|██████▎ | 297/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.22.linear_attn.conv1d.weight]
Loading weights: 63%|██████▎ | 298/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.22.linear_attn.dt_bias]
Loading weights: 63%|██████▎ | 298/473 [00:00<00:00, 408.43it/s, Materializing param=language_model.layers.22.linear_attn.dt_bias]
Loading weights: 63%|██████▎ | 299/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.dt_bias]
Loading weights: 63%|██████▎ | 299/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.in_proj_a.weight]
Loading weights: 63%|██████▎ | 299/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.in_proj_a.weight]
Loading weights: 63%|██████▎ | 300/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.in_proj_b.weight]
Loading weights: 63%|██████▎ | 300/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.in_proj_b.weight]
Loading weights: 64%|██████▎ | 301/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.in_proj_qkv.weight]
Loading weights: 64%|██████▎ | 301/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.in_proj_qkv.weight]
Loading weights: 64%|██████▍ | 302/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.in_proj_z.weight]
Loading weights: 64%|██████▍ | 302/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.in_proj_z.weight]
Loading weights: 64%|██████▍ | 303/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.norm.weight]
Loading weights: 64%|██████▍ | 303/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.norm.weight]
Loading weights: 64%|██████▍ | 304/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.out_proj.weight]
Loading weights: 64%|██████▍ | 304/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.linear_attn.out_proj.weight]
Loading weights: 64%|██████▍ | 305/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.mlp.down_proj.weight]
Loading weights: 64%|██████▍ | 305/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.mlp.down_proj.weight]
Loading weights: 65%|██████▍ | 306/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.mlp.gate_proj.weight]
Loading weights: 65%|██████▍ | 306/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.mlp.gate_proj.weight]
Loading weights: 65%|██████▍ | 307/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.mlp.up_proj.weight]
Loading weights: 65%|██████▍ | 307/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.mlp.up_proj.weight]
Loading weights: 65%|██████▌ | 308/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.post_attention_layernorm.weight]
Loading weights: 65%|██████▌ | 308/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.22.post_attention_layernorm.weight]
Loading weights: 65%|██████▌ | 309/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.input_layernorm.weight]
Loading weights: 65%|██████▌ | 309/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.input_layernorm.weight]
Loading weights: 66%|██████▌ | 310/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.mlp.down_proj.weight]
Loading weights: 66%|██████▌ | 310/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.mlp.down_proj.weight]
Loading weights: 66%|██████▌ | 311/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.mlp.gate_proj.weight]
Loading weights: 66%|██████▌ | 311/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.mlp.gate_proj.weight]
Loading weights: 66%|██████▌ | 312/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.mlp.up_proj.weight]
Loading weights: 66%|██████▌ | 312/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.mlp.up_proj.weight]
Loading weights: 66%|██████▌ | 313/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.post_attention_layernorm.weight]
Loading weights: 66%|██████▌ | 313/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.post_attention_layernorm.weight]
Loading weights: 66%|██████▋ | 314/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.k_norm.weight]
Loading weights: 66%|██████▋ | 314/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.k_norm.weight]
Loading weights: 67%|██████▋ | 315/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.k_proj.weight]
Loading weights: 67%|██████▋ | 315/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.k_proj.weight]
Loading weights: 67%|██████▋ | 316/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.o_proj.weight]
Loading weights: 67%|██████▋ | 316/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.o_proj.weight]
Loading weights: 67%|██████▋ | 317/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.q_norm.weight]
Loading weights: 67%|██████▋ | 317/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.q_norm.weight]
Loading weights: 67%|██████▋ | 318/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.q_proj.weight]
Loading weights: 67%|██████▋ | 318/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.q_proj.weight]
Loading weights: 67%|██████▋ | 319/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.v_proj.weight]
Loading weights: 67%|██████▋ | 319/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.layers.23.self_attn.v_proj.weight]
Loading weights: 68%|██████▊ | 320/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.norm.weight]
Loading weights: 68%|██████▊ | 320/473 [00:00<00:00, 366.23it/s, Materializing param=language_model.norm.weight]
Loading weights: 68%|██████▊ | 321/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.attn.proj.bias]
Loading weights: 68%|██████▊ | 321/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.attn.proj.bias]
Loading weights: 68%|██████▊ | 322/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.attn.proj.weight]
Loading weights: 68%|██████▊ | 322/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.attn.proj.weight]
Loading weights: 68%|██████▊ | 323/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.attn.qkv.bias]
Loading weights: 68%|██████▊ | 323/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.attn.qkv.bias]
Loading weights: 68%|██████▊ | 324/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.attn.qkv.weight]
Loading weights: 68%|██████▊ | 324/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.attn.qkv.weight]
Loading weights: 69%|██████▊ | 325/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.mlp.linear_fc1.bias]
Loading weights: 69%|██████▊ | 325/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.mlp.linear_fc1.bias]
Loading weights: 69%|██████▉ | 326/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.mlp.linear_fc1.weight]
Loading weights: 69%|██████▉ | 326/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.mlp.linear_fc1.weight]
Loading weights: 69%|██████▉ | 327/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.mlp.linear_fc2.bias]
Loading weights: 69%|██████▉ | 327/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.mlp.linear_fc2.bias]
Loading weights: 69%|██████▉ | 328/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.mlp.linear_fc2.weight]
Loading weights: 69%|██████▉ | 328/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.mlp.linear_fc2.weight]
Loading weights: 70%|██████▉ | 329/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.norm1.bias]
Loading weights: 70%|██████▉ | 329/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.norm1.bias]
Loading weights: 70%|██████▉ | 330/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.norm1.weight]
Loading weights: 70%|██████▉ | 330/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.norm1.weight]
Loading weights: 70%|██████▉ | 331/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.norm2.bias]
Loading weights: 70%|██████▉ | 331/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.norm2.bias]
Loading weights: 70%|███████ | 332/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.norm2.weight]
Loading weights: 70%|███████ | 332/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.0.norm2.weight]
Loading weights: 70%|███████ | 333/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.1.attn.proj.bias]
Loading weights: 70%|███████ | 333/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.1.attn.proj.bias]
Loading weights: 71%|███████ | 334/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.1.attn.proj.weight]
Loading weights: 71%|███████ | 334/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.1.attn.proj.weight]
Loading weights: 71%|███████ | 335/473 [00:00<00:00, 366.23it/s, Materializing param=visual.blocks.1.attn.qkv.bias]
Loading weights: 71%|███████ | 335/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.attn.qkv.bias]
Loading weights: 71%|███████ | 336/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.attn.qkv.weight]
Loading weights: 71%|███████ | 336/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.attn.qkv.weight]
Loading weights: 71%|███████ | 337/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.mlp.linear_fc1.bias]
Loading weights: 71%|███████ | 337/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.mlp.linear_fc1.bias]
Loading weights: 71%|███████▏ | 338/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.mlp.linear_fc1.weight]
Loading weights: 71%|███████▏ | 338/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.mlp.linear_fc1.weight]
Loading weights: 72%|███████▏ | 339/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.mlp.linear_fc2.bias]
Loading weights: 72%|███████▏ | 339/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.mlp.linear_fc2.bias]
Loading weights: 72%|███████▏ | 340/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.mlp.linear_fc2.weight]
Loading weights: 72%|███████▏ | 340/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.mlp.linear_fc2.weight]
Loading weights: 72%|███████▏ | 341/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.norm1.bias]
Loading weights: 72%|███████▏ | 341/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.norm1.bias]
Loading weights: 72%|███████▏ | 342/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.norm1.weight]
Loading weights: 72%|███████▏ | 342/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.norm1.weight]
Loading weights: 73%|███████▎ | 343/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.norm2.bias]
Loading weights: 73%|███████▎ | 343/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.norm2.bias]
Loading weights: 73%|███████▎ | 344/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.norm2.weight]
Loading weights: 73%|███████▎ | 344/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.1.norm2.weight]
Loading weights: 73%|███████▎ | 345/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.attn.proj.bias]
Loading weights: 73%|███████▎ | 345/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.attn.proj.bias]
Loading weights: 73%|███████▎ | 346/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.attn.proj.weight]
Loading weights: 73%|███████▎ | 346/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.attn.proj.weight]
Loading weights: 73%|███████▎ | 347/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.attn.qkv.bias]
Loading weights: 73%|███████▎ | 347/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.attn.qkv.bias]
Loading weights: 74%|███████▎ | 348/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.attn.qkv.weight]
Loading weights: 74%|███████▎ | 348/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.attn.qkv.weight]
Loading weights: 74%|███████▍ | 349/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.mlp.linear_fc1.bias]
Loading weights: 74%|███████▍ | 349/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.mlp.linear_fc1.bias]
Loading weights: 74%|███████▍ | 350/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.mlp.linear_fc1.weight]
Loading weights: 74%|███████▍ | 350/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.mlp.linear_fc1.weight]
Loading weights: 74%|███████▍ | 351/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.mlp.linear_fc2.bias]
Loading weights: 74%|███████▍ | 351/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.mlp.linear_fc2.bias]
Loading weights: 74%|███████▍ | 352/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.mlp.linear_fc2.weight]
Loading weights: 74%|███████▍ | 352/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.mlp.linear_fc2.weight]
Loading weights: 75%|███████▍ | 353/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.norm1.bias]
Loading weights: 75%|███████▍ | 353/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.norm1.bias]
Loading weights: 75%|███████▍ | 354/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.norm1.weight]
Loading weights: 75%|███████▍ | 354/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.norm1.weight]
Loading weights: 75%|███████▌ | 355/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.norm2.bias]
Loading weights: 75%|███████▌ | 355/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.norm2.bias]
Loading weights: 75%|███████▌ | 356/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.norm2.weight]
Loading weights: 75%|███████▌ | 356/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.2.norm2.weight]
Loading weights: 75%|███████▌ | 357/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.attn.proj.bias]
Loading weights: 75%|███████▌ | 357/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.attn.proj.bias]
Loading weights: 76%|███████▌ | 358/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.attn.proj.weight]
Loading weights: 76%|███████▌ | 358/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.attn.proj.weight]
Loading weights: 76%|███████▌ | 359/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.attn.qkv.bias]
Loading weights: 76%|███████▌ | 359/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.attn.qkv.bias]
Loading weights: 76%|███████▌ | 360/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.attn.qkv.weight]
Loading weights: 76%|███████▌ | 360/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.attn.qkv.weight]
Loading weights: 76%|███████▋ | 361/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.mlp.linear_fc1.bias]
Loading weights: 76%|███████▋ | 361/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.mlp.linear_fc1.bias]
Loading weights: 77%|███████▋ | 362/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.mlp.linear_fc1.weight]
Loading weights: 77%|███████▋ | 362/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.mlp.linear_fc1.weight]
Loading weights: 77%|███████▋ | 363/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.mlp.linear_fc2.bias]
Loading weights: 77%|███████▋ | 363/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.mlp.linear_fc2.bias]
Loading weights: 77%|███████▋ | 364/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.mlp.linear_fc2.weight]
Loading weights: 77%|███████▋ | 364/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.mlp.linear_fc2.weight]
Loading weights: 77%|███████▋ | 365/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.norm1.bias]
Loading weights: 77%|███████▋ | 365/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.norm1.bias]
Loading weights: 77%|███████▋ | 366/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.norm1.weight]
Loading weights: 77%|███████▋ | 366/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.norm1.weight]
Loading weights: 78%|███████▊ | 367/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.norm2.bias]
Loading weights: 78%|███████▊ | 367/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.norm2.bias]
Loading weights: 78%|███████▊ | 368/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.norm2.weight]
Loading weights: 78%|███████▊ | 368/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.3.norm2.weight]
Loading weights: 78%|███████▊ | 369/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.attn.proj.bias]
Loading weights: 78%|███████▊ | 369/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.attn.proj.bias]
Loading weights: 78%|███████▊ | 370/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.attn.proj.weight]
Loading weights: 78%|███████▊ | 370/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.attn.proj.weight]
Loading weights: 78%|███████▊ | 371/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.attn.qkv.bias]
Loading weights: 78%|███████▊ | 371/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.attn.qkv.bias]
Loading weights: 79%|███████▊ | 372/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.attn.qkv.weight]
Loading weights: 79%|███████▊ | 372/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.attn.qkv.weight]
Loading weights: 79%|███████▉ | 373/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.mlp.linear_fc1.bias]
Loading weights: 79%|███████▉ | 373/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.mlp.linear_fc1.bias]
Loading weights: 79%|███████▉ | 374/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.mlp.linear_fc1.weight]
Loading weights: 79%|███████▉ | 374/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.mlp.linear_fc1.weight]
Loading weights: 79%|███████▉ | 375/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.mlp.linear_fc2.bias]
Loading weights: 79%|███████▉ | 375/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.mlp.linear_fc2.bias]
Loading weights: 79%|███████▉ | 376/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.mlp.linear_fc2.weight]
Loading weights: 79%|███████▉ | 376/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.mlp.linear_fc2.weight]
Loading weights: 80%|███████▉ | 377/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.norm1.bias]
Loading weights: 80%|███████▉ | 377/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.norm1.bias]
Loading weights: 80%|███████▉ | 378/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.norm1.weight]
Loading weights: 80%|███████▉ | 378/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.norm1.weight]
Loading weights: 80%|████████ | 379/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.norm2.bias]
Loading weights: 80%|████████ | 379/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.norm2.bias]
Loading weights: 80%|████████ | 380/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.norm2.weight]
Loading weights: 80%|████████ | 380/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.4.norm2.weight]
Loading weights: 81%|████████ | 381/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.attn.proj.bias]
Loading weights: 81%|████████ | 381/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.attn.proj.bias]
Loading weights: 81%|████████ | 382/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.attn.proj.weight]
Loading weights: 81%|████████ | 382/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.attn.proj.weight]
Loading weights: 81%|████████ | 383/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.attn.qkv.bias]
Loading weights: 81%|████████ | 383/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.attn.qkv.bias]
Loading weights: 81%|████████ | 384/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.attn.qkv.weight]
Loading weights: 81%|████████ | 384/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.attn.qkv.weight]
Loading weights: 81%|████████▏ | 385/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.mlp.linear_fc1.bias]
Loading weights: 81%|████████▏ | 385/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.mlp.linear_fc1.bias]
Loading weights: 82%|████████▏ | 386/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.mlp.linear_fc1.weight]
Loading weights: 82%|████████▏ | 386/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.mlp.linear_fc1.weight]
Loading weights: 82%|████████▏ | 387/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.mlp.linear_fc2.bias]
Loading weights: 82%|████████▏ | 387/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.mlp.linear_fc2.bias]
Loading weights: 82%|████████▏ | 388/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.mlp.linear_fc2.weight]
Loading weights: 82%|████████▏ | 388/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.mlp.linear_fc2.weight]
Loading weights: 82%|████████▏ | 389/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.norm1.bias]
Loading weights: 82%|████████▏ | 389/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.norm1.bias]
Loading weights: 82%|████████▏ | 390/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.norm1.weight]
Loading weights: 82%|████████▏ | 390/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.norm1.weight]
Loading weights: 83%|████████▎ | 391/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.norm2.bias]
Loading weights: 83%|████████▎ | 391/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.norm2.bias]
Loading weights: 83%|████████▎ | 392/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.norm2.weight]
Loading weights: 83%|████████▎ | 392/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.5.norm2.weight]
Loading weights: 83%|████████▎ | 393/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.attn.proj.bias]
Loading weights: 83%|████████▎ | 393/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.attn.proj.bias]
Loading weights: 83%|████████▎ | 394/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.attn.proj.weight]
Loading weights: 83%|████████▎ | 394/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.attn.proj.weight]
Loading weights: 84%|████████▎ | 395/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.attn.qkv.bias]
Loading weights: 84%|████████▎ | 395/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.attn.qkv.bias]
Loading weights: 84%|████████▎ | 396/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.attn.qkv.weight]
Loading weights: 84%|████████▎ | 396/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.attn.qkv.weight]
Loading weights: 84%|████████▍ | 397/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.mlp.linear_fc1.bias]
Loading weights: 84%|████████▍ | 397/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.mlp.linear_fc1.bias]
Loading weights: 84%|████████▍ | 398/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.mlp.linear_fc1.weight]
Loading weights: 84%|████████▍ | 398/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.mlp.linear_fc1.weight]
Loading weights: 84%|████████▍ | 399/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.mlp.linear_fc2.bias]
Loading weights: 84%|████████▍ | 399/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.mlp.linear_fc2.bias]
Loading weights: 85%|████████▍ | 400/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.mlp.linear_fc2.weight]
Loading weights: 85%|████████▍ | 400/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.mlp.linear_fc2.weight]
Loading weights: 85%|████████▍ | 401/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.norm1.bias]
Loading weights: 85%|████████▍ | 401/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.norm1.bias]
Loading weights: 85%|████████▍ | 402/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.norm1.weight]
Loading weights: 85%|████████▍ | 402/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.norm1.weight]
Loading weights: 85%|████████▌ | 403/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.norm2.bias]
Loading weights: 85%|████████▌ | 403/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.norm2.bias]
Loading weights: 85%|████████▌ | 404/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.norm2.weight]
Loading weights: 85%|████████▌ | 404/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.6.norm2.weight]
Loading weights: 86%|████████▌ | 405/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.attn.proj.bias]
Loading weights: 86%|████████▌ | 405/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.attn.proj.bias]
Loading weights: 86%|████████▌ | 406/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.attn.proj.weight]
Loading weights: 86%|████████▌ | 406/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.attn.proj.weight]
Loading weights: 86%|████████▌ | 407/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.attn.qkv.bias]
Loading weights: 86%|████████▌ | 407/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.attn.qkv.bias]
Loading weights: 86%|████████▋ | 408/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.attn.qkv.weight]
Loading weights: 86%|████████▋ | 408/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.attn.qkv.weight]
Loading weights: 86%|████████▋ | 409/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.mlp.linear_fc1.bias]
Loading weights: 86%|████████▋ | 409/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.mlp.linear_fc1.bias]
Loading weights: 87%|████████▋ | 410/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.mlp.linear_fc1.weight]
Loading weights: 87%|████████▋ | 410/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.mlp.linear_fc1.weight]
Loading weights: 87%|████████▋ | 411/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.mlp.linear_fc2.bias]
Loading weights: 87%|████████▋ | 411/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.mlp.linear_fc2.bias]
Loading weights: 87%|████████▋ | 412/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.mlp.linear_fc2.weight]
Loading weights: 87%|████████▋ | 412/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.mlp.linear_fc2.weight]
Loading weights: 87%|████████▋ | 413/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.norm1.bias]
Loading weights: 87%|████████▋ | 413/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.norm1.bias]
Loading weights: 88%|████████▊ | 414/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.norm1.weight]
Loading weights: 88%|████████▊ | 414/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.norm1.weight]
Loading weights: 88%|████████▊ | 415/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.norm2.bias]
Loading weights: 88%|████████▊ | 415/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.norm2.bias]
Loading weights: 88%|████████▊ | 416/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.norm2.weight]
Loading weights: 88%|████████▊ | 416/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.7.norm2.weight]
Loading weights: 88%|████████▊ | 417/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.8.attn.proj.bias]
Loading weights: 88%|████████▊ | 417/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.8.attn.proj.bias]
Loading weights: 88%|████████▊ | 418/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.8.attn.proj.weight]
Loading weights: 88%|████████▊ | 418/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.8.attn.proj.weight]
Loading weights: 89%|████████▊ | 419/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.8.attn.qkv.bias]
Loading weights: 89%|████████▊ | 419/473 [00:01<00:00, 366.23it/s, Materializing param=visual.blocks.8.attn.qkv.bias]
Loading weights: 89%|████████▉ | 420/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.attn.qkv.bias]
Loading weights: 89%|████████▉ | 420/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.attn.qkv.weight]
Loading weights: 89%|████████▉ | 420/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.attn.qkv.weight]
Loading weights: 89%|████████▉ | 421/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.mlp.linear_fc1.bias]
Loading weights: 89%|████████▉ | 421/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.mlp.linear_fc1.bias]
Loading weights: 89%|████████▉ | 422/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.mlp.linear_fc1.weight]
Loading weights: 89%|████████▉ | 422/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.mlp.linear_fc1.weight]
Loading weights: 89%|████████▉ | 423/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.mlp.linear_fc2.bias]
Loading weights: 89%|████████▉ | 423/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.mlp.linear_fc2.bias]
Loading weights: 90%|████████▉ | 424/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.mlp.linear_fc2.weight]
Loading weights: 90%|████████▉ | 424/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.mlp.linear_fc2.weight]
Loading weights: 90%|████████▉ | 425/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.norm1.bias]
Loading weights: 90%|████████▉ | 425/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.norm1.bias]
Loading weights: 90%|█████████ | 426/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.norm1.weight]
Loading weights: 90%|█████████ | 426/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.norm1.weight]
Loading weights: 90%|█████████ | 427/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.norm2.bias]
Loading weights: 90%|█████████ | 427/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.norm2.bias]
Loading weights: 90%|█████████ | 428/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.norm2.weight]
Loading weights: 90%|█████████ | 428/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.8.norm2.weight]
Loading weights: 91%|█████████ | 429/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.attn.proj.bias]
Loading weights: 91%|█████████ | 429/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.attn.proj.bias]
Loading weights: 91%|█████████ | 430/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.attn.proj.weight]
Loading weights: 91%|█████████ | 430/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.attn.proj.weight]
Loading weights: 91%|█████████ | 431/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.attn.qkv.bias]
Loading weights: 91%|█████████ | 431/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.attn.qkv.bias]
Loading weights: 91%|█████████▏| 432/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.attn.qkv.weight]
Loading weights: 91%|█████████▏| 432/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.attn.qkv.weight]
Loading weights: 92%|█████████▏| 433/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.mlp.linear_fc1.bias]
Loading weights: 92%|█████████▏| 433/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.mlp.linear_fc1.bias]
Loading weights: 92%|█████████▏| 434/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.mlp.linear_fc1.weight]
Loading weights: 92%|█████████▏| 434/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.mlp.linear_fc1.weight]
Loading weights: 92%|█████████▏| 435/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.mlp.linear_fc2.bias]
Loading weights: 92%|█████████▏| 435/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.mlp.linear_fc2.bias]
Loading weights: 92%|█████████▏| 436/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.mlp.linear_fc2.weight]
Loading weights: 92%|█████████▏| 436/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.mlp.linear_fc2.weight]
Loading weights: 92%|█████████▏| 437/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.norm1.bias]
Loading weights: 92%|█████████▏| 437/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.norm1.bias]
Loading weights: 93%|█████████▎| 438/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.norm1.weight]
Loading weights: 93%|█████████▎| 438/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.norm1.weight]
Loading weights: 93%|█████████▎| 439/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.norm2.bias]
Loading weights: 93%|█████████▎| 439/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.norm2.bias]
Loading weights: 93%|█████████▎| 440/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.norm2.weight]
Loading weights: 93%|█████████▎| 440/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.9.norm2.weight]
Loading weights: 93%|█████████▎| 441/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.attn.proj.bias]
Loading weights: 93%|█████████▎| 441/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.attn.proj.bias]
Loading weights: 93%|█████████▎| 442/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.attn.proj.weight]
Loading weights: 93%|█████████▎| 442/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.attn.proj.weight]
Loading weights: 94%|█████████▎| 443/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.attn.qkv.bias]
Loading weights: 94%|█████████▎| 443/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.attn.qkv.bias]
Loading weights: 94%|█████████▍| 444/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.attn.qkv.weight]
Loading weights: 94%|█████████▍| 444/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.attn.qkv.weight]
Loading weights: 94%|█████████▍| 445/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.mlp.linear_fc1.bias]
Loading weights: 94%|█████████▍| 445/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.mlp.linear_fc1.bias]
Loading weights: 94%|█████████▍| 446/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.mlp.linear_fc1.weight]
Loading weights: 94%|█████████▍| 446/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.mlp.linear_fc1.weight]
Loading weights: 95%|█████████▍| 447/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.mlp.linear_fc2.bias]
Loading weights: 95%|█████████▍| 447/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.mlp.linear_fc2.bias]
Loading weights: 95%|█████████▍| 448/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.mlp.linear_fc2.weight]
Loading weights: 95%|█████████▍| 448/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.mlp.linear_fc2.weight]
Loading weights: 95%|█████████▍| 449/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.norm1.bias]
Loading weights: 95%|█████████▍| 449/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.norm1.bias]
Loading weights: 95%|█████████▌| 450/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.norm1.weight]
Loading weights: 95%|█████████▌| 450/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.norm1.weight]
Loading weights: 95%|█████████▌| 451/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.norm2.bias]
Loading weights: 95%|█████████▌| 451/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.norm2.bias]
Loading weights: 96%|█████████▌| 452/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.norm2.weight]
Loading weights: 96%|█████████▌| 452/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.10.norm2.weight]
Loading weights: 96%|█████████▌| 453/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.attn.proj.bias]
Loading weights: 96%|█████████▌| 453/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.attn.proj.bias]
Loading weights: 96%|█████████▌| 454/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.attn.proj.weight]
Loading weights: 96%|█████████▌| 454/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.attn.proj.weight]
Loading weights: 96%|█████████▌| 455/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.attn.qkv.bias]
Loading weights: 96%|█████████▌| 455/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.attn.qkv.bias]
Loading weights: 96%|█████████▋| 456/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.attn.qkv.weight]
Loading weights: 96%|█████████▋| 456/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.attn.qkv.weight]
Loading weights: 97%|█████████▋| 457/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.mlp.linear_fc1.bias]
Loading weights: 97%|█████████▋| 457/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.mlp.linear_fc1.bias]
Loading weights: 97%|█████████▋| 458/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.mlp.linear_fc1.weight]
Loading weights: 97%|█████████▋| 458/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.mlp.linear_fc1.weight]
Loading weights: 97%|█████████▋| 459/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.mlp.linear_fc2.bias]
Loading weights: 97%|█████████▋| 459/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.mlp.linear_fc2.bias]
Loading weights: 97%|█████████▋| 460/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.mlp.linear_fc2.weight]
Loading weights: 97%|█████████▋| 460/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.mlp.linear_fc2.weight]
Loading weights: 97%|█████████▋| 461/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.norm1.bias]
Loading weights: 97%|█████████▋| 461/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.norm1.bias]
Loading weights: 98%|█████████▊| 462/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.norm1.weight]
Loading weights: 98%|█████████▊| 462/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.norm1.weight]
Loading weights: 98%|█████████▊| 463/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.norm2.bias]
Loading weights: 98%|█████████▊| 463/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.norm2.bias]
Loading weights: 98%|█████████▊| 464/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.norm2.weight]
Loading weights: 98%|█████████▊| 464/473 [00:01<00:00, 594.62it/s, Materializing param=visual.blocks.11.norm2.weight]
Loading weights: 98%|█████████▊| 465/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.linear_fc1.bias]
Loading weights: 98%|█████████▊| 465/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.linear_fc1.bias]
Loading weights: 99%|█████████▊| 466/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.linear_fc1.weight]
Loading weights: 99%|█████████▊| 466/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.linear_fc1.weight]
Loading weights: 99%|█████████▊| 467/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.linear_fc2.bias]
Loading weights: 99%|█████████▊| 467/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.linear_fc2.bias]
Loading weights: 99%|█████████▉| 468/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.linear_fc2.weight]
Loading weights: 99%|█████████▉| 468/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.linear_fc2.weight]
Loading weights: 99%|█████████▉| 469/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.norm.bias]
Loading weights: 99%|█████████▉| 469/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.norm.bias]
Loading weights: 99%|█████████▉| 470/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.norm.weight]
Loading weights: 99%|█████████▉| 470/473 [00:01<00:00, 594.62it/s, Materializing param=visual.merger.norm.weight]
Loading weights: 100%|█████████▉| 471/473 [00:01<00:00, 594.62it/s, Materializing param=visual.patch_embed.proj.bias]
Loading weights: 100%|█████████▉| 471/473 [00:01<00:00, 594.62it/s, Materializing param=visual.patch_embed.proj.bias]
Loading weights: 100%|█████████▉| 472/473 [00:01<00:00, 594.62it/s, Materializing param=visual.patch_embed.proj.weight]
Loading weights: 100%|█████████▉| 472/473 [00:01<00:00, 594.62it/s, Materializing param=visual.patch_embed.proj.weight]
Loading weights: 100%|██████████| 473/473 [00:01<00:00, 594.62it/s, Materializing param=visual.pos_embed.weight]
Loading weights: 100%|██████████| 473/473 [00:01<00:00, 594.62it/s, Materializing param=visual.pos_embed.weight]
Loading weights: 100%|██████████| 473/473 [00:01<00:00, 449.91it/s, Materializing param=visual.pos_embed.weight]
Teacher loaded: 853.0M params
Loading student from MVP/artifacts/qwen35-0.8b-text-n4...
Loading weights: 0%| | 0/55 [00:00<?, ?it/s]
Loading weights: 2%|▏ | 1/55 [00:00<00:00, 30393.51it/s, Materializing param=model.embed_tokens.weight]
Loading weights: 2%|▏ | 1/55 [00:00<00:00, 6563.86it/s, Materializing param=model.embed_tokens.weight]
Loading weights: 4%|▎ | 2/55 [00:00<00:09, 5.60it/s, Materializing param=model.embed_tokens.weight]
Loading weights: 4%|▎ | 2/55 [00:00<00:09, 5.60it/s, Materializing param=model.layers.0.input_layernorm.weight]
Loading weights: 4%|▎ | 2/55 [00:00<00:09, 5.60it/s, Materializing param=model.layers.0.input_layernorm.weight]
Loading weights: 5%|▌ | 3/55 [00:00<00:09, 5.60it/s, Materializing param=model.layers.0.linear_attn.A_log]
Loading weights: 5%|▌ | 3/55 [00:00<00:09, 5.60it/s, Materializing param=model.layers.0.linear_attn.A_log]
Loading weights: 7%|▋ | 4/55 [00:00<00:09, 5.60it/s, Materializing param=model.layers.0.linear_attn.conv1d.weight]
Loading weights: 7%|▋ | 4/55 [00:00<00:09, 5.60it/s, Materializing param=model.layers.0.linear_attn.conv1d.weight]
Loading weights: 9%|▉ | 5/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.dt_bias]
Loading weights: 9%|▉ | 5/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.dt_bias]
Loading weights: 11%|█ | 6/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.in_proj_a.weight]
Loading weights: 11%|█ | 6/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.in_proj_a.weight]
Loading weights: 13%|█▎ | 7/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.in_proj_b.weight]
Loading weights: 13%|█▎ | 7/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.in_proj_b.weight]
Loading weights: 15%|█▍ | 8/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.in_proj_qkv.weight]
Loading weights: 15%|█▍ | 8/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.in_proj_qkv.weight]
Loading weights: 16%|█▋ | 9/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.in_proj_z.weight]
Loading weights: 16%|█▋ | 9/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.in_proj_z.weight]
Loading weights: 18%|█▊ | 10/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.norm.weight]
Loading weights: 18%|█▊ | 10/55 [00:00<00:08, 5.60it/s, Materializing param=model.layers.0.linear_attn.norm.weight]
Loading weights: 20%|██ | 11/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.linear_attn.out_proj.weight]
Loading weights: 20%|██ | 11/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.linear_attn.out_proj.weight]
Loading weights: 22%|██▏ | 12/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.mlp.down_proj.weight]
Loading weights: 22%|██▏ | 12/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.mlp.down_proj.weight]
Loading weights: 24%|██▎ | 13/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.mlp.gate_proj.weight]
Loading weights: 24%|██▎ | 13/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.mlp.gate_proj.weight]
Loading weights: 25%|██▌ | 14/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.mlp.up_proj.weight]
Loading weights: 25%|██▌ | 14/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.mlp.up_proj.weight]
Loading weights: 27%|██▋ | 15/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.post_attention_layernorm.weight]
Loading weights: 27%|██▋ | 15/55 [00:00<00:07, 5.60it/s, Materializing param=model.layers.0.post_attention_layernorm.weight]
Loading weights: 29%|██▉ | 16/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.input_layernorm.weight]
Loading weights: 29%|██▉ | 16/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.input_layernorm.weight]
Loading weights: 31%|███ | 17/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.A_log]
Loading weights: 31%|███ | 17/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.A_log]
Loading weights: 33%|███▎ | 18/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.conv1d.weight]
Loading weights: 33%|███▎ | 18/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.conv1d.weight]
Loading weights: 35%|███▍ | 19/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.dt_bias]
Loading weights: 35%|███▍ | 19/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.dt_bias]
Loading weights: 36%|███▋ | 20/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.in_proj_a.weight]
Loading weights: 36%|███▋ | 20/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.in_proj_a.weight]
Loading weights: 38%|███▊ | 21/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.in_proj_b.weight]
Loading weights: 38%|███▊ | 21/55 [00:00<00:06, 5.60it/s, Materializing param=model.layers.1.linear_attn.in_proj_b.weight]
Loading weights: 40%|████ | 22/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.linear_attn.in_proj_qkv.weight]
Loading weights: 40%|████ | 22/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.linear_attn.in_proj_qkv.weight]
Loading weights: 42%|████▏ | 23/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.linear_attn.in_proj_z.weight]
Loading weights: 42%|████▏ | 23/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.linear_attn.in_proj_z.weight]
Loading weights: 44%|████▎ | 24/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.linear_attn.norm.weight]
Loading weights: 44%|████▎ | 24/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.linear_attn.norm.weight]
Loading weights: 45%|████▌ | 25/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.linear_attn.out_proj.weight]
Loading weights: 45%|████▌ | 25/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.linear_attn.out_proj.weight]
Loading weights: 47%|████▋ | 26/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.mlp.down_proj.weight]
Loading weights: 47%|████▋ | 26/55 [00:00<00:05, 5.60it/s, Materializing param=model.layers.1.mlp.down_proj.weight]
Loading weights: 49%|████▉ | 27/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.1.mlp.gate_proj.weight]
Loading weights: 49%|████▉ | 27/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.1.mlp.gate_proj.weight]
Loading weights: 51%|█████ | 28/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.1.mlp.up_proj.weight]
Loading weights: 51%|█████ | 28/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.1.mlp.up_proj.weight]
Loading weights: 53%|█████▎ | 29/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.1.post_attention_layernorm.weight]
Loading weights: 53%|█████▎ | 29/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.1.post_attention_layernorm.weight]
Loading weights: 55%|█████▍ | 30/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.2.input_layernorm.weight]
Loading weights: 55%|█████▍ | 30/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.2.input_layernorm.weight]
Loading weights: 56%|█████▋ | 31/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.2.linear_attn.A_log]
Loading weights: 56%|█████▋ | 31/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.2.linear_attn.A_log]
Loading weights: 58%|█████▊ | 32/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.2.linear_attn.conv1d.weight]
Loading weights: 58%|█████▊ | 32/55 [00:00<00:04, 5.60it/s, Materializing param=model.layers.2.linear_attn.conv1d.weight]
Loading weights: 60%|██████ | 33/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.dt_bias]
Loading weights: 60%|██████ | 33/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.dt_bias]
Loading weights: 62%|██████▏ | 34/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.in_proj_a.weight]
Loading weights: 62%|██████▏ | 34/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.in_proj_a.weight]
Loading weights: 64%|██████▎ | 35/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.in_proj_b.weight]
Loading weights: 64%|██████▎ | 35/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.in_proj_b.weight]
Loading weights: 65%|██████▌ | 36/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.in_proj_qkv.weight]
Loading weights: 65%|██████▌ | 36/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.in_proj_qkv.weight]
Loading weights: 67%|██████▋ | 37/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.in_proj_z.weight]
Loading weights: 67%|██████▋ | 37/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.in_proj_z.weight]
Loading weights: 69%|██████▉ | 38/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.norm.weight]
Loading weights: 69%|██████▉ | 38/55 [00:00<00:03, 5.60it/s, Materializing param=model.layers.2.linear_attn.norm.weight]
Loading weights: 71%|███████ | 39/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.linear_attn.out_proj.weight]
Loading weights: 71%|███████ | 39/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.linear_attn.out_proj.weight]
Loading weights: 73%|███████▎ | 40/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.mlp.down_proj.weight]
Loading weights: 73%|███████▎ | 40/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.mlp.down_proj.weight]
Loading weights: 75%|███████▍ | 41/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.mlp.gate_proj.weight]
Loading weights: 75%|███████▍ | 41/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.mlp.gate_proj.weight]
Loading weights: 76%|███████▋ | 42/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.mlp.up_proj.weight]
Loading weights: 76%|███████▋ | 42/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.mlp.up_proj.weight]
Loading weights: 78%|███████▊ | 43/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.post_attention_layernorm.weight]
Loading weights: 78%|███████▊ | 43/55 [00:00<00:02, 5.60it/s, Materializing param=model.layers.2.post_attention_layernorm.weight]
Loading weights: 80%|████████ | 44/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.input_layernorm.weight]
Loading weights: 80%|████████ | 44/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.input_layernorm.weight]
Loading weights: 82%|████████▏ | 45/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.mlp.down_proj.weight]
Loading weights: 82%|████████▏ | 45/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.mlp.down_proj.weight]
Loading weights: 84%|████████▎ | 46/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.mlp.gate_proj.weight]
Loading weights: 84%|████████▎ | 46/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.mlp.gate_proj.weight]
Loading weights: 85%|████████▌ | 47/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.mlp.up_proj.weight]
Loading weights: 85%|████████▌ | 47/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.mlp.up_proj.weight]
Loading weights: 87%|████████▋ | 48/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.post_attention_layernorm.weight]
Loading weights: 87%|████████▋ | 48/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.post_attention_layernorm.weight]
Loading weights: 89%|████████▉ | 49/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.self_attn.k_norm.weight]
Loading weights: 89%|████████▉ | 49/55 [00:00<00:01, 5.60it/s, Materializing param=model.layers.3.self_attn.k_norm.weight]
Loading weights: 91%|█████████ | 50/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.k_proj.weight]
Loading weights: 91%|█████████ | 50/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.k_proj.weight]
Loading weights: 93%|█████████▎| 51/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.o_proj.weight]
Loading weights: 93%|█████████▎| 51/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.o_proj.weight]
Loading weights: 95%|█████████▍| 52/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.q_norm.weight]
Loading weights: 95%|█████████▍| 52/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.q_norm.weight]
Loading weights: 96%|█████████▋| 53/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.q_proj.weight]
Loading weights: 96%|█████████▋| 53/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.q_proj.weight]
Loading weights: 98%|█████████▊| 54/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.v_proj.weight]
Loading weights: 98%|█████████▊| 54/55 [00:00<00:00, 5.60it/s, Materializing param=model.layers.3.self_attn.v_proj.weight]
Loading weights: 100%|██████████| 55/55 [00:00<00:00, 5.60it/s, Materializing param=model.norm.weight]
Loading weights: 100%|██████████| 55/55 [00:00<00:00, 5.60it/s, Materializing param=model.norm.weight]
Loading weights: 100%|██████████| 55/55 [00:00<00:00, 149.22it/s, Materializing param=model.norm.weight]
The tokenizer you are loading from 'MVP/artifacts/qwen35-0.8b-text-n4' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
Token indices sequence length is longer than the specified maximum sequence length for this model (2505798 > Student loaded: 337.3M params
Loading tokenizer...
Tokenizing dataset (/tmp/wik/usr/lib/python3.14/multiprocessing/resource_tracker.py:475: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {'/mp-z_hczt5i'}
warnings.warn(
en=128
Step 25/2000 | total=288.4217 ce=13.6517 mse=27.477000 | lr=1.50e-05 | 107s elapsed, ~8492s remaining
Step 50/2000 | total=249.2028 ce=12.1995 mse=23.700326 | lr=3.00e-05 | 214s elapsed, ~8336s remaining
Step 75/2000 | total=182.6658 ce=9.3742 mse=17.329160 | lr=4.50e-05 | 320s elapsed, ~8217s remaining
Step 100/2000 | total=138.5065 ce=7.9229 mse=13.058353 | lr=6.00e-05 | 427s elapsed, ~8108s remaining
Step 125/2000 | total=115.7030 ce=7.5666 mse=10.813636 | lr=7.50e-05 | 532s elapsed, ~7977s remaining
Step 150/2000 | total=106.5914 ce=7.3182 mse=9.927319 | lr=9.00e-05 | 638s elapsed, ~7865s remaining
Step 175/2000 | total=100.8314 ce=6.9378 mse=9.389356 | lr=1.05e-04 | 743s elapsed, ~7747s remaining
Step 200/2000 | total=98.4073 ce=6.5909 mse=9.181641 | lr=1.25e-04 | 850s elapsed, ~7648s remaining
Step 225/2000 | total=93.9046 ce=6.3370 mse=8.756758 | lr=1.40e-04 | 955s elapsed, ~7536s remaining
Step 250/2000 | total=88.4071 ce=6.0484 mse=8.235866 | lr=1.55e-04 | 1539s elapsed, ~10774s remaining
Traceback (most recent call last):
File "/home/void0x14/Documents/echo/MVP/distill.py", line 289, in <module>
_main()
~~~~~^^
File "/home/void0x14/Documents/echo/MVP/distill.py", line 285, in _main
run_distill(DistillConfig(**{k.replace("-", "_"): v for k, v in vars(args).items()}))
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/void0x14/Documents/echo/MVP/distill.py", line 256, in run_distill
save_checkpoint(student, student_config, args.output_dir, step + 1)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/void0x14/Documents/echo/MVP/distill.py", line 160, in save_checkpoint
config_path.write_text(json.dumps(student_config.to_dict(), indent=2, sort_keys=True) + "\n", encoding="utf-8")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.14/json/__init__.py", line 242, in dumps
**kw).encode(obj)
~~~~~~^^^^^
File "/usr/lib/python3.14/json/encoder.py", line 202, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.14/json/encoder.py", line 263, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.14/json/encoder.py", line 182, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
f'is not JSON serializable')
TypeError: Object of type Qwen3_5TextConfig is not JSON serializable
|