Upload logs/p1_shipper.log with huggingface_hub
Browse files- logs/p1_shipper.log +21 -0
logs/p1_shipper.log
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[14:05:07] === shipper armed: FREQ=200, full resumable at: 599 600 1199 1200 ===
|
| 2 |
+
[14:27:42] step 200 -> rank_0 only (bisection point)
|
| 3 |
+
VERIFIED checkpoints/p1_680m_step200.ckpt local=5,141,244,514 remote=5141244514
|
| 4 |
+
[14:28:24] step 200 verified on HF
|
| 5 |
+
[14:53:59] step 400 -> rank_0 only (bisection point)
|
| 6 |
+
VERIFIED checkpoints/p1_680m_step400.ckpt local=5,141,244,514 remote=5141244514
|
| 7 |
+
[14:54:45] step 400 verified on HF
|
| 8 |
+
[15:20:20] step 600 -> FULL resumable (all ranks, .ckpt + .zero_g1)
|
| 9 |
+
VERIFIED checkpoints/p1_680m_step600_rank0.ckpt local=5,141,244,514 remote=5141244514
|
| 10 |
+
VERIFIED checkpoints/p1_680m_step600_rank0.zero_g1 local=464,090,276 remote=464090276
|
| 11 |
+
VERIFIED checkpoints/p1_680m_step600_rank1.ckpt local=5,141,244,514 remote=5141244514
|
| 12 |
+
VERIFIED checkpoints/p1_680m_step600_rank1.zero_g1 local=464,090,276 remote=464090276
|
| 13 |
+
[15:22:20] step 600 FULL verified on HF
|
| 14 |
+
[15:47:11] step 800 -> rank_0 only (bisection point)
|
| 15 |
+
VERIFIED checkpoints/p1_680m_step800.ckpt local=5,141,244,514 remote=5141244514
|
| 16 |
+
[15:48:02] step 800 verified on HF
|
| 17 |
+
[16:12:52] step 1000 -> rank_0 only (bisection point)
|
| 18 |
+
VERIFIED checkpoints/p1_680m_step1000.ckpt local=5,141,244,514 remote=5141244514
|
| 19 |
+
[16:15:39] step 1000 verified on HF
|
| 20 |
+
[16:39:44] step 1200 -> FULL resumable (all ranks, .ckpt + .zero_g1)
|
| 21 |
+
VERIFIED checkpoints/p1_680m_step1200_rank0.ckpt local=5,141,244,514 remote=5141244514
|