Configuration (bluscriptCP):
  d_model: 1536
  n_layers: 36
  q_heads: 12
  kv_heads: 4  (G=3, head_dim=128)
  ffn_hidden: 4096
  context_length: 4096
  vocab_size: 50304  (tying=0)
  B: 4
  T: 4096
  world_size: 4  (dp=4 x cp=1)
  CP_ATTN_MODE: ring  (load_balance=HeadTail)
  global_batch: 1048576
  grad_accum_steps: 16
  Parameters: 1060624896
  Parameters per GPU: 1060624896
  Max Learning Rate: 0.001
  Min Learning Rate: 0.0001
  max_steps: 1000
  warmup_steps: 100
  checkpointing: 1
  ckpt_freq: 25
  run: 8
  GPU Memory Used (rank 0): 20906.9 MB / 143155.9 MB