File size: 552 Bytes
d7922f9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | Configuration (bluscriptCP): d_model: 1536 n_layers: 36 q_heads: 12 kv_heads: 4 (G=3, head_dim=128) ffn_hidden: 4096 context_length: 16384 vocab_size: 50304 (tying=0) B: 1 T: 16384 world_size: 4 (dp=1 x cp=4) CP_ATTN_MODE: ring (load_balance=HeadTail) global_batch: 1048576 grad_accum_steps: 64 Parameters: 1060624896 Parameters per GPU: 1060624896 Max Learning Rate: 0.001 Min Learning Rate: 0.0001 max_steps: 4 warmup_steps: 1 checkpointing: 0 run: 2 GPU Memory Used (rank 0): 20904.9 MB / 143155.9 MB |