unparallelled commited on
Commit
c518028
·
verified ·
1 Parent(s): 4a9a7da

Upload folder using huggingface_hub

Browse files
logs/retrain_corrected_cache/arm_a.ckptpath ADDED
@@ -0,0 +1 @@
 
 
1
+ DEFERRED-not-trained (arm A intentionally skipped 2026-09-03; run it later with: rm retrain_logs/arm_a.ckptpath && ./retrain_corrected_cache.sh)
logs/retrain_corrected_cache/arm_b.log ADDED
The diff for this file is too large to render. See raw diff
 
logs/retrain_corrected_cache/arm_b.runnum ADDED
@@ -0,0 +1 @@
 
 
1
+ 7
logs/retrain_corrected_cache/driver.out ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ [07:19:19] free disk 92G (need ~70G: 4 checkpoint sets at ~17G)
2
+ [07:19:19] arm data ready
3
+ [07:19:19] run7' already done (checkpoints_bluscriptcp/rank_0/blumodelcp_run6_step_750.ckpt)
4
+ [07:19:19] arm a already done
5
+ [07:19:19] training arm b (corrected cache)
logs/retrain_corrected_cache/master.log ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [04:40:01] free disk 116G (need ~70G: 4 checkpoint sets at ~17G)
2
+ [04:40:01] generating arm a data
3
+ wrote 3 shard(s) in in2_data/arm_a 209977344 tokens total
4
+ answer tokens 0 (0.00%) synthetic 0 (0.00%)
5
+ wrote 1 shard(s) in in2_data/arm_a 1048576 tokens total
6
+ answer tokens 0 (0.00%) synthetic 0 (0.00%)
7
+ [04:40:09] generating arm b data
8
+ wrote 3 shard(s) in in2_data/arm_b 209977344 tokens total
9
+ answer tokens 6561792 (3.12%) synthetic 22146048 (10.55%)
10
+ distance: min 12 median 12 max 12 n=820224
11
+ wrote 1 shard(s) in in2_data/arm_b 1048576 tokens total
12
+ answer tokens 32768 (3.12%) synthetic 110592 (10.55%)
13
+ distance: min 12 median 12 max 12 n=4096
14
+ [04:40:37] generating arm c data
15
+ wrote 3 shard(s) in in2_data/arm_c 209977344 tokens total
16
+ answer tokens 6561792 (3.12%) synthetic 22146048 (10.55%)
17
+ distance: min 76 median 2923 max 16350 n=820224
18
+ wrote 1 shard(s) in in2_data/arm_c 1048576 tokens total
19
+ answer tokens 32768 (3.12%) synthetic 110592 (10.55%)
20
+ distance: min 76 median 2921 max 16194 n=4096
21
+ [04:41:05] arm data ready
22
+ [04:41:05] training run7' (YaRN 16k extension, corrected cache)
23
+ [07:14:12] run7' (YaRN 16k extension, corrected cache) -> run 6, blumodelcp_run6_step_750.ckpt, final loss loss: 2.383854
24
+ [07:14:12] arm a already done
25
+ [07:14:12] training arm b (corrected cache)
26
+ [07:14:59] FATAL: arm b (corrected cache) training failed -- see retrain_logs/arm_b.log
27
+ [07:19:19] free disk 92G (need ~70G: 4 checkpoint sets at ~17G)
28
+ [07:19:19] arm data ready
29
+ [07:19:19] run7' already done (checkpoints_bluscriptcp/rank_0/blumodelcp_run6_step_750.ckpt)
30
+ [07:19:19] arm a already done
31
+ [07:19:19] training arm b (corrected cache)
logs/retrain_corrected_cache/run7c.ckptpath ADDED
@@ -0,0 +1 @@
 
 
1
+ checkpoints_bluscriptcp/rank_0/blumodelcp_run6_step_750.ckpt
logs/retrain_corrected_cache/run7c.log ADDED
The diff for this file is too large to render. See raw diff