Upload docs/TEARDOWN_READY_l3long_2026-09-09.txt with huggingface_hub
Browse files
docs/TEARDOWN_READY_l3long_2026-09-09.txt
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEARDOWN READY -- 2026-09-09 18:35:01 UTC
|
| 2 |
+
|
| 3 |
+
Run 17 (l3long) STOPPED at step 2450 of 4000. GPUs released.
|
| 4 |
+
All results, logs, scripts, source snapshot and checkpoints verified on
|
| 5 |
+
unparallelled/BluScriptCP by local-vs-remote byte size.
|
| 6 |
+
|
| 7 |
+
UPLOADED
|
| 8 |
+
docs/RESULTS_l3long_2026-09-09.md eval tables + disk-leak writeup
|
| 9 |
+
docs/session_2026-09-09/ session report + obsidian log line
|
| 10 |
+
logs/l3long_run17/ training log + janitor log
|
| 11 |
+
logs/l3long_eval1000/ , l3long_eval2000/ both NIAH eval ladders
|
| 12 |
+
scripts/ b3_long_run, eval_at_1000/2000,
|
| 13 |
+
ext_32k, ckpt_janitor
|
| 14 |
+
source/src_context_parallelism_2026-09-09.tgz 123 source files, 7.0 MB
|
| 15 |
+
source/src_blutrain_root_2026-09-09.tgz root .cpp/.cu/.py
|
| 16 |
+
checkpoints/l3long1000_run17_step_1000_rank{0,1}.ckpt
|
| 17 |
+
checkpoints/l3long2000_run17_step_2000_rank{0,1}.ckpt
|
| 18 |
+
checkpoints/l3long_final_run17_step_2450_rank{0,1}.ckpt
|
| 19 |
+
checkpoints/l3long_final_run17_step_2450_rank{0,1}.zero_g1 <- optimizer
|
| 20 |
+
state, so step 2450 is RESUMABLE
|
| 21 |
+
|
| 22 |
+
TO RESUME l3long LATER
|
| 23 |
+
Pull the four step_2450 files, restore them as
|
| 24 |
+
checkpoints_bluscriptcp/rank_N/blumodelcp_run17_step_2450.{ckpt,zero_g1}
|
| 25 |
+
then ./b3_long_run.sh resume 17 (it picks the newest run17 ckpt)
|
| 26 |
+
Remaining: 1550 steps at 11.56 s/step = ~5.0 h. Decay begins at step 3900.
|
| 27 |
+
|
| 28 |
+
STILL NOT DONE
|
| 29 |
+
1. Decay check: STEPS=1100 STABLE=900 from the step-1000 ckpt.
|
| 30 |
+
2. ext_32k.sh (parent l2, b3 data) -- written, never run. NOTE a DIFFERENT
|
| 31 |
+
32k extension (ext32k_b4, b4 data) WAS run and evaluated earlier today;
|
| 32 |
+
see docs/EXT32K_RESULTS_2026-09-09.md and logs/ext32k/.
|
| 33 |
+
3. bluscript_amp1.cpp and the dist/Context_Parallelism submodule were left
|
| 34 |
+
MODIFIED and UNCOMMITTED. I cannot run git per project rules. The source
|
| 35 |
+
tarballs above are the only backup -- commit them yourself before relying
|
| 36 |
+
on git history.
|
| 37 |
+
|
| 38 |
+
NOT UPLOADED (deliberate, re-derivable)
|
| 39 |
+
lendata/ 3.3 GB regenerate with fetch_eval_assets.sh
|
| 40 |
+
build/ rebuild with make
|
| 41 |
+
intermediate run17 checkpoints (KEEP=2 pruned them during the run)
|