Upload folder using huggingface_hub
Browse files- logs/retrain_corrected_cache/arm_a.ckptpath +1 -0
- logs/retrain_corrected_cache/arm_b.log +0 -0
- logs/retrain_corrected_cache/arm_b.runnum +1 -0
- logs/retrain_corrected_cache/driver.out +5 -0
- logs/retrain_corrected_cache/master.log +31 -0
- logs/retrain_corrected_cache/run7c.ckptpath +1 -0
- logs/retrain_corrected_cache/run7c.log +0 -0
logs/retrain_corrected_cache/arm_a.ckptpath
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
DEFERRED-not-trained (arm A intentionally skipped 2026-09-03; run it later with: rm retrain_logs/arm_a.ckptpath && ./retrain_corrected_cache.sh)
|
logs/retrain_corrected_cache/arm_b.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
logs/retrain_corrected_cache/arm_b.runnum
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
7
|
logs/retrain_corrected_cache/driver.out
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[07:19:19] free disk 92G (need ~70G: 4 checkpoint sets at ~17G)
|
| 2 |
+
[07:19:19] arm data ready
|
| 3 |
+
[07:19:19] run7' already done (checkpoints_bluscriptcp/rank_0/blumodelcp_run6_step_750.ckpt)
|
| 4 |
+
[07:19:19] arm a already done
|
| 5 |
+
[07:19:19] training arm b (corrected cache)
|
logs/retrain_corrected_cache/master.log
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[04:40:01] free disk 116G (need ~70G: 4 checkpoint sets at ~17G)
|
| 2 |
+
[04:40:01] generating arm a data
|
| 3 |
+
wrote 3 shard(s) in in2_data/arm_a 209977344 tokens total
|
| 4 |
+
answer tokens 0 (0.00%) synthetic 0 (0.00%)
|
| 5 |
+
wrote 1 shard(s) in in2_data/arm_a 1048576 tokens total
|
| 6 |
+
answer tokens 0 (0.00%) synthetic 0 (0.00%)
|
| 7 |
+
[04:40:09] generating arm b data
|
| 8 |
+
wrote 3 shard(s) in in2_data/arm_b 209977344 tokens total
|
| 9 |
+
answer tokens 6561792 (3.12%) synthetic 22146048 (10.55%)
|
| 10 |
+
distance: min 12 median 12 max 12 n=820224
|
| 11 |
+
wrote 1 shard(s) in in2_data/arm_b 1048576 tokens total
|
| 12 |
+
answer tokens 32768 (3.12%) synthetic 110592 (10.55%)
|
| 13 |
+
distance: min 12 median 12 max 12 n=4096
|
| 14 |
+
[04:40:37] generating arm c data
|
| 15 |
+
wrote 3 shard(s) in in2_data/arm_c 209977344 tokens total
|
| 16 |
+
answer tokens 6561792 (3.12%) synthetic 22146048 (10.55%)
|
| 17 |
+
distance: min 76 median 2923 max 16350 n=820224
|
| 18 |
+
wrote 1 shard(s) in in2_data/arm_c 1048576 tokens total
|
| 19 |
+
answer tokens 32768 (3.12%) synthetic 110592 (10.55%)
|
| 20 |
+
distance: min 76 median 2921 max 16194 n=4096
|
| 21 |
+
[04:41:05] arm data ready
|
| 22 |
+
[04:41:05] training run7' (YaRN 16k extension, corrected cache)
|
| 23 |
+
[07:14:12] run7' (YaRN 16k extension, corrected cache) -> run 6, blumodelcp_run6_step_750.ckpt, final loss loss: 2.383854
|
| 24 |
+
[07:14:12] arm a already done
|
| 25 |
+
[07:14:12] training arm b (corrected cache)
|
| 26 |
+
[07:14:59] FATAL: arm b (corrected cache) training failed -- see retrain_logs/arm_b.log
|
| 27 |
+
[07:19:19] free disk 92G (need ~70G: 4 checkpoint sets at ~17G)
|
| 28 |
+
[07:19:19] arm data ready
|
| 29 |
+
[07:19:19] run7' already done (checkpoints_bluscriptcp/rank_0/blumodelcp_run6_step_750.ckpt)
|
| 30 |
+
[07:19:19] arm a already done
|
| 31 |
+
[07:19:19] training arm b (corrected cache)
|
logs/retrain_corrected_cache/run7c.ckptpath
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
checkpoints_bluscriptcp/rank_0/blumodelcp_run6_step_750.ckpt
|
logs/retrain_corrected_cache/run7c.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|