quantization_metric/metrics.log

nohup: ignoring input

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:18<00:18, 18.79s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 11.66s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 12.73s/it]
Once upon a time, there was a man who was born with a birthmark on his face. He was born with a birthmark that was a perfect circle that resembled a moon.
His name was Jack.
Jack was born with a birthmark that was a perfect circle that resembled a moon.
The birthmark was so perfect that people were fascinated by it. They would always ask Jack about it.
Jack would tell them that it was a birthmark.
He
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (v_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pad_token_id": 0,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.55.2",
  "use_cache": true,
  "vocab_size": 32000
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 0 ---1.5555332899093628 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 1 ---2.2105441093444824 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 2 ---2.6319708824157715 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 3 ---2.659501791000366 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 4 ---2.5770697593688965 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 5 ---2.5436081886291504 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 6 ---2.4908900260925293 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 7 ---2.5257110595703125 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 8 ---2.3653383255004883 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 9 ---2.5174360275268555 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 10 ---2.265111207962036 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 11 ---2.1847732067108154 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 12 ---2.4449100494384766 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 13 ---2.679959774017334 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 14 ---2.4503092765808105 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 15 ---2.7230710983276367 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 16 ---3.074552536010742 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 17 ---3.4709739685058594 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 18 ---3.67897629737854 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 19 ---3.278068780899048 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 20 ---3.6138486862182617 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 21 ---3.5603649616241455 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 22 ---3.9758076667785645 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 23 ---4.087326526641846 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 24 ---3.739630699157715 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 25 ---4.076397895812988 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 26 ---3.5009336471557617 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 27 ---4.056451320648193 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 28 ---3.726351737976074 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 29 ---3.844115972518921 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 30 ---4.4837751388549805 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 31 ---3.275714874267578 
metric_name alpha: [30, 23, 25, 27, 22, 29, 24, 28, 18, 20, 21, 26, 17, 19, 31, 16, 15, 13, 3, 2, 4, 5, 7, 9, 6, 14, 12, 8, 10, 1, 11, 0]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:18<00:18, 18.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.16s/it]
Once upon a time, there was a little girl who was the light of her parents’ eyes. She had a younger brother who was the apple of her father’s eye. She was a good student who loved math and science. She was a good athlete who played sports, and she loved to sing and dance.
She was a good girl who was kind and loving and caring. She was a good girl who was always there for her friends. She was a good girl who loved her parents and
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (v_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pad_token_id": 0,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.55.2",
  "use_cache": true,
  "vocab_size": 32000
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 0 ---6.557916641235352 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 1 ---8.015230178833008 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 2 ---10.61414909362793 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 3 ---10.561344146728516 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 4 ---10.057601928710938 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 5 ---9.833120346069336 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 6 ---9.45051097869873 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 7 ---9.582311630249023 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 8 ---9.064985275268555 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 9 ---9.556177139282227 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 10 ---8.45679759979248 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 11 ---8.441352844238281 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 12 ---9.276637077331543 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 13 ---10.002967834472656 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 14 ---8.847436904907227 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 15 ---9.9490327835083 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 16 ---11.152729988098145 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 17 ---12.680035591125488 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 18 ---13.309869766235352 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 19 ---12.054608345031738 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 20 ---13.724580764770508 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 21 ---13.702856063842773 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 22 ---15.829018592834473 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 23 ---15.232747077941895 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 24 ---14.636650085449219 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 25 ---15.008004188537598 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 26 ---14.163816452026367 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 27 ---15.760412216186523 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 28 ---14.682222366333008 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 29 ---15.929686546325684 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 30 ---17.405780792236328 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 31 ---14.380212783813477 
metric_name alpha_hat: [30, 29, 22, 27, 23, 25, 28, 24, 31, 26, 20, 21, 18, 17, 19, 16, 2, 3, 4, 13, 15, 5, 7, 9, 6, 12, 8, 14, 10, 11, 1, 0]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:18<00:18, 18.48s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 11.47s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 12.52s/it]
Once upon a time, in a land far away, there lived a prince who had everything money could buy, including a beautiful wife, an amazing mansion, and a huge car collection. The prince loved his cars, but one day he decided to take his car collection to the next level.
He hired a company to build him a huge garage to house his collection. The garage was built to be the biggest and best in the world. It had state-of-the-art security systems
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (v_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pad_token_id": 0,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.55.2",
  "use_cache": true,
  "vocab_size": 32000
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(55.2111, device='cuda:0')
spectral_norm tensor(21.8310, device='cuda:0')
frobenius_norm tensor(61.8584, device='cuda:0')
spectral_norm tensor(18.5959, device='cuda:0')
frobenius_norm tensor(45.2757, device='cuda:0')
spectral_norm tensor(4.0318, device='cuda:0')
frobenius_norm tensor(29.4097, device='cuda:0')
spectral_norm tensor(4.3865, device='cuda:0')
alpha value of layer 0 ---47.129005432128906 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(108.3667, device='cuda:0')
spectral_norm tensor(18.4210, device='cuda:0')
frobenius_norm tensor(108.1698, device='cuda:0')
spectral_norm tensor(20.1468, device='cuda:0')
frobenius_norm tensor(41.1449, device='cuda:0')
spectral_norm tensor(3.2622, device='cuda:0')
frobenius_norm tensor(33.7843, device='cuda:0')
spectral_norm tensor(3.8698, device='cuda:0')
alpha value of layer 1 ---74.68388366699219 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(109.0373, device='cuda:0')
spectral_norm tensor(16.2222, device='cuda:0')
frobenius_norm tensor(114.6870, device='cuda:0')
spectral_norm tensor(19.2055, device='cuda:0')
frobenius_norm tensor(58.7294, device='cuda:0')
spectral_norm tensor(3.5355, device='cuda:0')
frobenius_norm tensor(56.9624, device='cuda:0')
spectral_norm tensor(6.0202, device='cuda:0')
alpha value of layer 2 ---111.57455444335938 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(103.1725, device='cuda:0')
spectral_norm tensor(13.4197, device='cuda:0')
frobenius_norm tensor(107.3360, device='cuda:0')
spectral_norm tensor(15.3238, device='cuda:0')
frobenius_norm tensor(55.7174, device='cuda:0')
spectral_norm tensor(3.0726, device='cuda:0')
frobenius_norm tensor(54.5137, device='cuda:0')
spectral_norm tensor(6.3542, device='cuda:0')
alpha value of layer 3 ---127.65095520019531 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(107.5970, device='cuda:0')
spectral_norm tensor(13.7911, device='cuda:0')
frobenius_norm tensor(109.8770, device='cuda:0')
spectral_norm tensor(15.7544, device='cuda:0')
frobenius_norm tensor(58.8042, device='cuda:0')
spectral_norm tensor(3.1027, device='cuda:0')
frobenius_norm tensor(57.5345, device='cuda:0')
spectral_norm tensor(5.8239, device='cuda:0')
alpha value of layer 4 ---141.5764617919922 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(108.1569, device='cuda:0')
spectral_norm tensor(13.7375, device='cuda:0')
frobenius_norm tensor(112.6002, device='cuda:0')
spectral_norm tensor(16.5425, device='cuda:0')
frobenius_norm tensor(60.3288, device='cuda:0')
spectral_norm tensor(2.9971, device='cuda:0')
frobenius_norm tensor(59.0258, device='cuda:0')
spectral_norm tensor(5.6560, device='cuda:0')
alpha value of layer 5 ---155.60284423828125 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(102.0697, device='cuda:0')
spectral_norm tensor(12.2985, device='cuda:0')
frobenius_norm tensor(104.0983, device='cuda:0')
spectral_norm tensor(14.5729, device='cuda:0')
frobenius_norm tensor(55.9676, device='cuda:0')
spectral_norm tensor(3.0027, device='cuda:0')
frobenius_norm tensor(55.1871, device='cuda:0')
spectral_norm tensor(5.7670, device='cuda:0')
alpha value of layer 6 ---139.72598266601562 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(101.8819, device='cuda:0')
spectral_norm tensor(11.7685, device='cuda:0')
frobenius_norm tensor(102.3675, device='cuda:0')
spectral_norm tensor(13.6839, device='cuda:0')
frobenius_norm tensor(56.6824, device='cuda:0')
spectral_norm tensor(3.1391, device='cuda:0')
frobenius_norm tensor(55.6199, device='cuda:0')
spectral_norm tensor(5.4573, device='cuda:0')
alpha value of layer 7 ---140.21083068847656 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(102.8848, device='cuda:0')
spectral_norm tensor(11.9707, device='cuda:0')
frobenius_norm tensor(103.4811, device='cuda:0')
spectral_norm tensor(14.2754, device='cuda:0')
frobenius_norm tensor(58.2330, device='cuda:0')
spectral_norm tensor(3.3746, device='cuda:0')
frobenius_norm tensor(57.2962, device='cuda:0')
spectral_norm tensor(4.9391, device='cuda:0')
alpha value of layer 8 ---139.6942138671875 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(103.0146, device='cuda:0')
spectral_norm tensor(12.2318, device='cuda:0')
frobenius_norm tensor(105.5969, device='cuda:0')
spectral_norm tensor(14.2079, device='cuda:0')
frobenius_norm tensor(59.3876, device='cuda:0')
spectral_norm tensor(3.2388, device='cuda:0')
frobenius_norm tensor(58.5812, device='cuda:0')
spectral_norm tensor(5.1232, device='cuda:0')
alpha value of layer 9 ---148.2813262939453 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(102.8745, device='cuda:0')
spectral_norm tensor(12.0922, device='cuda:0')
frobenius_norm tensor(106.0223, device='cuda:0')
spectral_norm tensor(14.3113, device='cuda:0')
frobenius_norm tensor(58.7986, device='cuda:0')
spectral_norm tensor(3.2255, device='cuda:0')
frobenius_norm tensor(58.3338, device='cuda:0')
spectral_norm tensor(4.3048, device='cuda:0')
alpha value of layer 10 ---160.80075073242188 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(97.7702, device='cuda:0')
spectral_norm tensor(11.1815, device='cuda:0')
frobenius_norm tensor(97.4910, device='cuda:0')
spectral_norm tensor(12.9324, device='cuda:0')
frobenius_norm tensor(61.3144, device='cuda:0')
spectral_norm tensor(3.4012, device='cuda:0')
frobenius_norm tensor(60.7354, device='cuda:0')
spectral_norm tensor(5.6488, device='cuda:0')
alpha value of layer 11 ---143.46920776367188 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(99.7752, device='cuda:0')
spectral_norm tensor(11.8016, device='cuda:0')
frobenius_norm tensor(102.8686, device='cuda:0')
spectral_norm tensor(13.5998, device='cuda:0')
frobenius_norm tensor(60.5482, device='cuda:0')
spectral_norm tensor(3.2452, device='cuda:0')
frobenius_norm tensor(60.0323, device='cuda:0')
spectral_norm tensor(4.9509, device='cuda:0')
alpha value of layer 12 ---155.95849609375 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(98.8230, device='cuda:0')
spectral_norm tensor(12.0874, device='cuda:0')
frobenius_norm tensor(100.6162, device='cuda:0')
spectral_norm tensor(13.8355, device='cuda:0')
frobenius_norm tensor(62.7593, device='cuda:0')
spectral_norm tensor(3.1144, device='cuda:0')
frobenius_norm tensor(62.1430, device='cuda:0')
spectral_norm tensor(5.1164, device='cuda:0')
alpha value of layer 13 ---168.32928466796875 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(98.5708, device='cuda:0')
spectral_norm tensor(11.6312, device='cuda:0')
frobenius_norm tensor(100.4783, device='cuda:0')
spectral_norm tensor(13.6670, device='cuda:0')
frobenius_norm tensor(61.6071, device='cuda:0')
spectral_norm tensor(2.7584, device='cuda:0')
frobenius_norm tensor(60.9149, device='cuda:0')
spectral_norm tensor(4.5464, device='cuda:0')
alpha value of layer 14 ---201.04998779296875 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(97.3649, device='cuda:0')
spectral_norm tensor(12.7196, device='cuda:0')
frobenius_norm tensor(100.7325, device='cuda:0')
spectral_norm tensor(14.3247, device='cuda:0')
frobenius_norm tensor(64.0580, device='cuda:0')
spectral_norm tensor(3.0007, device='cuda:0')
frobenius_norm tensor(63.2220, device='cuda:0')
spectral_norm tensor(4.5882, device='cuda:0')
alpha value of layer 15 ---188.4088134765625 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(96.6512, device='cuda:0')
spectral_norm tensor(12.9655, device='cuda:0')
frobenius_norm tensor(99.2465, device='cuda:0')
spectral_norm tensor(14.6775, device='cuda:0')
frobenius_norm tensor(66.7600, device='cuda:0')
spectral_norm tensor(2.8352, device='cuda:0')
frobenius_norm tensor(66.0842, device='cuda:0')
spectral_norm tensor(5.0123, device='cuda:0')
alpha value of layer 16 ---207.397216796875 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(95.8736, device='cuda:0')
spectral_norm tensor(12.8995, device='cuda:0')
frobenius_norm tensor(98.0118, device='cuda:0')
spectral_norm tensor(14.3731, device='cuda:0')
frobenius_norm tensor(66.5281, device='cuda:0')
spectral_norm tensor(2.8901, device='cuda:0')
frobenius_norm tensor(66.1344, device='cuda:0')
spectral_norm tensor(5.4531, device='cuda:0')
alpha value of layer 17 ---194.68035888671875 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(93.7199, device='cuda:0')
spectral_norm tensor(12.9969, device='cuda:0')
frobenius_norm tensor(95.7889, device='cuda:0')
spectral_norm tensor(14.0707, device='cuda:0')
frobenius_norm tensor(69.6604, device='cuda:0')
spectral_norm tensor(2.8885, device='cuda:0')
frobenius_norm tensor(68.6924, device='cuda:0')
spectral_norm tensor(5.4377, device='cuda:0')
alpha value of layer 18 ---209.88125610351562 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(92.5769, device='cuda:0')
spectral_norm tensor(12.7937, device='cuda:0')
frobenius_norm tensor(94.3567, device='cuda:0')
spectral_norm tensor(14.2169, device='cuda:0')
frobenius_norm tensor(70.2688, device='cuda:0')
spectral_norm tensor(2.7430, device='cuda:0')
frobenius_norm tensor(69.5499, device='cuda:0')
spectral_norm tensor(5.3632, device='cuda:0')
alpha value of layer 19 ---230.21255493164062 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(93.1738, device='cuda:0')
spectral_norm tensor(13.3309, device='cuda:0')
frobenius_norm tensor(94.8772, device='cuda:0')
spectral_norm tensor(14.2162, device='cuda:0')
frobenius_norm tensor(71.3496, device='cuda:0')
spectral_norm tensor(2.7475, device='cuda:0')
frobenius_norm tensor(70.9048, device='cuda:0')
spectral_norm tensor(6.5838, device='cuda:0')
alpha value of layer 20 ---220.93441772460938 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(91.0892, device='cuda:0')
spectral_norm tensor(12.7824, device='cuda:0')
frobenius_norm tensor(92.0887, device='cuda:0')
spectral_norm tensor(13.3415, device='cuda:0')
frobenius_norm tensor(73.5470, device='cuda:0')
spectral_norm tensor(3.2134, device='cuda:0')
frobenius_norm tensor(72.4853, device='cuda:0')
spectral_norm tensor(5.6293, device='cuda:0')
alpha value of layer 21 ---197.01531982421875 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(93.0198, device='cuda:0')
spectral_norm tensor(12.9645, device='cuda:0')
frobenius_norm tensor(94.1975, device='cuda:0')
spectral_norm tensor(13.3901, device='cuda:0')
frobenius_norm tensor(73.8086, device='cuda:0')
spectral_norm tensor(2.7246, device='cuda:0')
frobenius_norm tensor(72.6579, device='cuda:0')
spectral_norm tensor(7.5949, device='cuda:0')
alpha value of layer 22 ---231.5908660888672 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(92.3679, device='cuda:0')
spectral_norm tensor(12.5056, device='cuda:0')
frobenius_norm tensor(93.0806, device='cuda:0')
spectral_norm tensor(12.8793, device='cuda:0')
frobenius_norm tensor(77.2716, device='cuda:0')
spectral_norm tensor(3.0203, device='cuda:0')
frobenius_norm tensor(76.3245, device='cuda:0')
spectral_norm tensor(5.5035, device='cuda:0')
alpha value of layer 23 ---238.41143798828125 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(89.8033, device='cuda:0')
spectral_norm tensor(12.3866, device='cuda:0')
frobenius_norm tensor(90.2768, device='cuda:0')
spectral_norm tensor(13.1727, device='cuda:0')
frobenius_norm tensor(76.5770, device='cuda:0')
spectral_norm tensor(3.2186, device='cuda:0')
frobenius_norm tensor(75.2567, device='cuda:0')
spectral_norm tensor(6.6725, device='cuda:0')
alpha value of layer 24 ---198.1973419189453 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(90.5618, device='cuda:0')
spectral_norm tensor(11.8575, device='cuda:0')
frobenius_norm tensor(90.8598, device='cuda:0')
spectral_norm tensor(12.2949, device='cuda:0')
frobenius_norm tensor(79.4490, device='cuda:0')
spectral_norm tensor(3.1827, device='cuda:0')
frobenius_norm tensor(78.3357, device='cuda:0')
spectral_norm tensor(4.8220, device='cuda:0')
alpha value of layer 25 ---249.99850463867188 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(89.1381, device='cuda:0')
spectral_norm tensor(12.8790, device='cuda:0')
frobenius_norm tensor(89.8395, device='cuda:0')
spectral_norm tensor(13.2880, device='cuda:0')
frobenius_norm tensor(80.7266, device='cuda:0')
spectral_norm tensor(3.6409, device='cuda:0')
frobenius_norm tensor(80.1935, device='cuda:0')
spectral_norm tensor(6.8510, device='cuda:0')
alpha value of layer 26 ---180.56016540527344 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(92.4526, device='cuda:0')
spectral_norm tensor(13.1710, device='cuda:0')
frobenius_norm tensor(93.3447, device='cuda:0')
spectral_norm tensor(14.0394, device='cuda:0')
frobenius_norm tensor(80.8654, device='cuda:0')
spectral_norm tensor(3.3240, device='cuda:0')
frobenius_norm tensor(80.7260, device='cuda:0')
spectral_norm tensor(5.6484, device='cuda:0')
alpha value of layer 27 ---222.39707946777344 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(89.8511, device='cuda:0')
spectral_norm tensor(12.7679, device='cuda:0')
frobenius_norm tensor(90.9173, device='cuda:0')
spectral_norm tensor(13.5210, device='cuda:0')
frobenius_norm tensor(83.4357, device='cuda:0')
spectral_norm tensor(3.6919, device='cuda:0')
frobenius_norm tensor(83.0720, device='cuda:0')
spectral_norm tensor(6.0824, device='cuda:0')
alpha value of layer 28 ---198.00421142578125 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(87.5255, device='cuda:0')
spectral_norm tensor(13.2835, device='cuda:0')
frobenius_norm tensor(88.2859, device='cuda:0')
spectral_norm tensor(14.1804, device='cuda:0')
frobenius_norm tensor(83.7624, device='cuda:0')
spectral_norm tensor(4.7727, device='cuda:0')
frobenius_norm tensor(84.0506, device='cuda:0')
spectral_norm tensor(6.8564, device='cuda:0')
alpha value of layer 29 ---135.1178436279297 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(88.0865, device='cuda:0')
spectral_norm tensor(13.0963, device='cuda:0')
frobenius_norm tensor(89.2752, device='cuda:0')
spectral_norm tensor(13.7996, device='cuda:0')
frobenius_norm tensor(85.7229, device='cuda:0')
spectral_norm tensor(3.7462, device='cuda:0')
frobenius_norm tensor(86.1523, device='cuda:0')
spectral_norm tensor(6.8602, device='cuda:0')
alpha value of layer 30 ---192.10064697265625 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
frobenius_norm tensor(89.1405, device='cuda:0')
spectral_norm tensor(15.0259, device='cuda:0')
frobenius_norm tensor(92.3933, device='cuda:0')
spectral_norm tensor(16.3841, device='cuda:0')
frobenius_norm tensor(78.1290, device='cuda:0')
spectral_norm tensor(4.0098, device='cuda:0')
frobenius_norm tensor(78.9173, device='cuda:0')
spectral_norm tensor(10.5689, device='cuda:0')
alpha value of layer 31 ---125.60063171386719 
metric_name stable_rank: [25, 23, 22, 19, 27, 20, 18, 16, 14, 24, 28, 21, 17, 30, 15, 26, 13, 10, 12, 5, 9, 11, 4, 7, 6, 8, 29, 3, 31, 2, 1, 0]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:18<00:18, 18.13s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.18s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.22s/it]
Once upon a time, there was a very rich man who had a very beautiful daughter. Once upon a time, there was a very rich man who had a very beautiful daughter. The rich man’s name was King Midas.
The rich man’s name was King Midas. The rich man’s name was King Midas. The rich man’s name was King Midas. The rich man’s name was King Midas. The rich man’s name was King Midas.

LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (v_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pad_token_id": 0,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.55.2",
  "use_cache": true,
  "vocab_size": 32000
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 0 ---1640.3033447265625 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 1 ---2134.1572265625 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 2 ---2669.992919921875 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 3 ---2901.25439453125 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 4 ---2904.2001953125 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 5 ---2910.534912109375 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 6 ---2883.722900390625 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 7 ---2887.236328125 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 8 ---2899.039306640625 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 9 ---2916.92822265625 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 10 ---2859.56689453125 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 11 ---2818.8173828125 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 12 ---2905.6064453125 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 13 ---2940.74462890625 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 14 ---2900.401123046875 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 15 ---2949.82080078125 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 16 ---2976.977783203125 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 17 ---3047.2646484375 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 18 ---3096.2216796875 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 19 ---3061.852783203125 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 20 ---3062.37353515625 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 21 ---3081.3349609375 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 22 ---3106.181640625 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 23 ---3144.513427734375 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 24 ---3072.8798828125 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 25 ---3137.80224609375 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 26 ---3090.37158203125 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 27 ---3181.7998046875 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 28 ---3147.865478515625 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 29 ---3101.146484375 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 30 ---3161.5263671875 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 31 ---3049.5556640625 
metric_name effective_rank: [27, 30, 28, 23, 25, 22, 29, 18, 26, 21, 24, 20, 19, 31, 17, 16, 15, 13, 9, 5, 12, 4, 3, 14, 8, 7, 6, 10, 11, 2, 1, 0]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:18<00:18, 18.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.13s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.16s/it]
Once upon a time, there was a princess who was born with a pearl in her mouth. She would be the queen of the world.
Once upon a time, there was a princess who was born with a pearl in her mouth. She would be the queen of the world. This story is a fable about the birth of the princess, who is named after the pearl in her mouth. The princess’s mother was a beautiful, wise, and strong woman
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (v_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pad_token_id": 0,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.55.2",
  "use_cache": true,
  "vocab_size": 32000
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 0 ---0.09540334343910217 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 1 ---0.11126542091369629 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 2 ---0.14089055359363556 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 3 ---0.1446058303117752 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 4 ---0.14712807536125183 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 5 ---0.1478433907032013 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 6 ---0.14464625716209412 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 7 ---0.14459004998207092 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 8 ---0.14641690254211426 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 9 ---0.147793248295784 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 10 ---0.14709556102752686 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 11 ---0.14403118193149567 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 12 ---0.14700128138065338 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 13 ---0.1479380875825882 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 14 ---0.1479010283946991 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 15 ---0.1490364670753479 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 16 ---0.1480296403169632 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 17 ---0.15020982921123505 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 18 ---0.1507750302553177 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 19 ---0.14981798827648163 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 20 ---0.15018826723098755 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 21 ---0.1498291790485382 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 22 ---0.15043966472148895 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 23 ---0.151978999376297 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 24 ---0.14919137954711914 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 25 ---0.15175150334835052 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 26 ---0.1495654433965683 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 27 ---0.15338149666786194 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 28 ---0.15180033445358276 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 29 ---0.1501537710428238 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 30 ---0.15218497812747955 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 31 ---0.14980709552764893 
metric_name ZD: [27, 30, 23, 28, 25, 18, 22, 17, 20, 29, 21, 19, 31, 26, 24, 15, 16, 13, 14, 5, 9, 4, 10, 12, 8, 6, 3, 7, 11, 2, 1, 0]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:18<00:18, 18.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.13s/it]
Once upon a time, there lived a wealthy man who had a son whom he loved dearly. He was very proud of his son and loved to show him off to everyone who came to visit.
One day, a young man came to visit. He was a good friend of the son. He was very poor and did not have a home to live in. He asked if he could stay the night. The man’s son said, “Of course, I’ll let you stay.”
The
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (v_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pad_token_id": 0,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.55.2",
  "use_cache": true,
  "vocab_size": 32000
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 0 ---0.9916330575942993 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 1 ---0.9952021241188049 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 2 ---0.9966323971748352 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 3 ---0.9973293542861938 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 4 ---0.9971895217895508 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 5 ---0.9973934888839722 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 6 ---0.9974462389945984 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 7 ---0.9975071549415588 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 8 ---0.9974231719970703 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 9 ---0.9973534345626831 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 10 ---0.997123122215271 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 11 ---0.9970043897628784 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 12 ---0.9973783493041992 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 13 ---0.9974591732025146 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 14 ---0.9971306324005127 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 15 ---0.9973533153533936 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 16 ---0.9974291324615479 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 17 ---0.9976841807365417 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 18 ---0.997740626335144 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 19 ---0.9975850582122803 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 20 ---0.9973828792572021 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 21 ---0.9975684881210327 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 22 ---0.9977440237998962 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 23 ---0.9980273246765137 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 24 ---0.9974839091300964 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 25 ---0.9979180693626404 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 26 ---0.9974991083145142 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 27 ---0.9979188442230225 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 28 ---0.997989296913147 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 29 ---0.9974175691604614 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 30 ---0.9975640773773193 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 31 ---0.997219443321228 
metric_name head_diversity: [23, 28, 27, 25, 22, 18, 17, 19, 21, 30, 7, 26, 24, 13, 6, 16, 8, 29, 5, 20, 12, 9, 15, 3, 31, 4, 14, 10, 11, 2, 1, 0]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:18<00:18, 18.55s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 11.50s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 12.55s/it]
Once upon a time, there was a little girl who was born on July 23, 1991, in the city of Chicago, Illinois. She had a good childhood, despite her parents getting divorced when she was young. She had a lot of friends and loved to sing and dance. She was very good at both. When she was 10, she started taking singing lessons. Her voice was very beautiful and she was soon discovered by a record company. They signed her and
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (v_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pad_token_id": 0,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.55.2",
  "use_cache": true,
  "vocab_size": 32000
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 0 ---0.08510372042655945 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 1 ---0.04102545976638794 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 2 ---0.028616365045309067 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 3 ---0.02104165218770504 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 4 ---0.022063206881284714 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 5 ---0.021188031882047653 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 6 ---0.020417138934135437 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 7 ---0.019520433619618416 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 8 ---0.020254574716091156 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 9 ---0.020007748156785965 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 10 ---0.021119512617588043 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 11 ---0.020985007286071777 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 12 ---0.019723106175661087 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 13 ---0.01894117146730423 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 14 ---0.01963678002357483 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 15 ---0.01925666816532612 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 16 ---0.018222851678729057 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 17 ---0.016996942460536957 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 18 ---0.016209837049245834 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 19 ---0.017241276800632477 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 20 ---0.017154088243842125 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 21 ---0.016598742455244064 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 22 ---0.016119930893182755 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 23 ---0.015261407010257244 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 24 ---0.01685335859656334 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 25 ---0.015361565165221691 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 26 ---0.01685093343257904 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 27 ---0.015206292271614075 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 28 ---0.01575298234820366 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 29 ---0.01735319383442402 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 30 ---0.016395289450883865 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)}
alpha value of layer 31 ---0.02029731497168541 
metric_name coherence: [0, 1, 2, 4, 5, 10, 3, 11, 6, 31, 8, 9, 12, 14, 7, 15, 13, 16, 29, 19, 20, 17, 24, 26, 21, 30, 18, 22, 28, 25, 23, 27]