File size: 544 Bytes
7625259 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | Configuration (bluscriptCP): d_model: 1536 n_layers: 24 q_heads: 12 kv_heads: 4 (G=3, head_dim=128) ffn_hidden: 4096 context_length: 4096 vocab_size: 50304 (tying=1) B: 1 T: 4096 world_size: 4 (dp=2 x cp=2) CP_ATTN_MODE: ring (load_balance=HeadTail) global_batch: 8192 grad_accum_steps: 1 Parameters: 681328128 Parameters per GPU: 681328128 Max Learning Rate: 0.0006 Min Learning Rate: 6e-05 max_steps: 1 warmup_steps: 91 checkpointing: 0 run: 2 GPU Memory Used (rank 0): 6068.9 MB / 143155.9 MB |