File size: 571 Bytes
7625259 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | Configuration (bluscriptCP): d_model: 1536 n_layers: 36 q_heads: 12 kv_heads: 4 (G=3, head_dim=128) ffn_hidden: 4096 context_length: 4096 vocab_size: 50304 (tying=0) B: 2 T: 4096 world_size: 4 (dp=4 x cp=1) CP_ATTN_MODE: ring (load_balance=HeadTail) global_batch: 1048576 grad_accum_steps: 32 Parameters: 1060624896 Parameters per GPU: 1060624896 Max Learning Rate: 0.0006 Min Learning Rate: 6e-05 max_steps: 1000 warmup_steps: 100 checkpointing: 1 ckpt_freq: 25 run: 7 GPU Memory Used (rank 0): 20906.9 MB / 143155.9 MB |