File size: 550 Bytes
7625259 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | Configuration (bluscriptCP): d_model: 1536 n_layers: 36 q_heads: 12 kv_heads: 4 (G=3, head_dim=128) ffn_hidden: 4096 context_length: 4096 vocab_size: 50304 (tying=0) B: 4 T: 4096 world_size: 4 (dp=4 x cp=1) CP_ATTN_MODE: ring (load_balance=HeadTail) global_batch: 1048576 grad_accum_steps: 16 Parameters: 1060624896 Parameters per GPU: 1060624896 Max Learning Rate: 0.0006 Min Learning Rate: 6e-05 max_steps: 3 warmup_steps: 1 checkpointing: 0 run: 5 GPU Memory Used (rank 0): 20906.9 MB / 143155.9 MB |