| Configuration (bluscriptCP): | |
| d_model: 1536 | |
| n_layers: 24 | |
| q_heads: 12 | |
| kv_heads: 4 (G=3, head_dim=128) | |
| ffn_hidden: 4096 | |
| context_length: 4096 | |
| vocab_size: 50304 (tying=1) | |
| B: 1 | |
| T: 4096 | |
| world_size: 4 (dp=2 x cp=2) | |
| CP_ATTN_MODE: ring (load_balance=HeadTail) | |
| global_batch: 8192 | |
| grad_accum_steps: 1 | |
| Parameters: 681328128 | |
| Parameters per GPU: 681328128 | |
| Max Learning Rate: 0.0006 | |
| Min Learning Rate: 6e-05 | |
| max_steps: 1 | |
| warmup_steps: 91 | |
| checkpointing: 0 | |
| run: 2 | |
| GPU Memory Used (rank 0): 6068.9 MB / 143155.9 MB | |