Upload folder using huggingface_hub

docs/session_2026-09-09/ClaudeReport_session_2026-09-09_ckpt_disk_leak_and_eval2000.md

@@ -0,0 +1,60 @@
+# Session 2026-09-09 - checkpoint disk leak, janitor, step-2000 eval
+
+2026-09-09 - Fixed silent checkpoint failure from disk exhaustion, added sidecar janitor, armed step-2000 NIAH eval - Context_Parallelism / ckpt_janitor.sh, eval_at_2000.sh
+
+## Root cause found
+
+`CP_CKPT_KEEP=2` prunes `.ckpt` files but NOT their `.zero_g1` optimizer sidecars.
+At `CP_CKPT_FREQ=25` this leaked ~54 orphaned sidecars per rank (0.46 GB each,
+~49 GB total) and filled the 193 GB disk.
+
+Consequence: every checkpoint save from step 1275 onward failed for ~75 minutes
+while training carried on normally, because the trainer catches a failed save and
+only emits `[WARN] checkpoint save at step NNNN failed`. No crash, no backpressure.
+
+Freed 63 GB (14.3 GB abandoned `.tmp` + the orphaned sidecars). Saves healthy
+again, confirmed at step 1650.
+
+Cost: no valid checkpoint between step 1250 and step 1650. Nothing lost since the
+run survived, but the run was unprotected for ~400 steps.
+
+## Mitigation
+
+`ckpt_janitor.sh` - detached, 5-minute loop. Deletes `.zero_g1` sidecars with no
+surviving `.ckpt`, removes `.tmp` older than 10 min, warns to
+`b3long_logs/janitor.log` if free space drops under 25 GB (a save needs 17.3 GB).
+
+## LR schedule - resolved, not a bug
+
+Observed flat `1.8000e-04` from step 100 onward was correct all along:
+`CP_REWARMUP_PEAK=0.3 x max_lr 6e-4 = 1.8e-4`. An earlier session note predicted
+3.0e-4 by assuming max_lr=1e-3; the actual configured max_lr is 6e-4.
+Banner: `warmup/steps 100/4000 (WSD: stable=3800, decay tail=100)` - so decay
+does not begin until step 3900. Every mid-run eval is on an undecayed model.
+
+## NIAH single-needle, step-1000 checkpoint (accuracy %)
+
+| prompt_len | armA_c | l1 | l2 | l3 | l3long@1000 |
+|---|---|---|---|---|---|
+| 4096  | 73.0 | 75.0 | 80.0 | 74.0 | 65.0 |
+| 8192  | 53.0 | 48.0 | 62.0 | 64.0 | 55.0 |
+| 16360 | 33.0 | 30.0 | 35.0 | 35.0 | 36.0 |
+
+l3long is worst at 4096 (-9 vs l3), mid at 8192 (-9 vs l3), and best at 16360 by
+a single point (+1 vs l3/l2). The 16k-training premise is not yet paying off.
+Caveat that cuts in l3long's favour: l3long@1000 is undecayed, l3@400 is decayed.
+
+## State at session end
+
+Run 17 -> step 4000, ~11.58 s/step, loss 2.61 at step 1690, ETA ~23:05 UTC.
+Detached (PPID 1, survive session close): `b3_long_run.sh resume 17`,
+`ckpt_janitor.sh`, `eval_at_2000.sh`.
+
+`eval_at_2000.sh` stops the trainer at step 2000, preserves the ckpt as
+`l3long2000_*`, evals armA_c/l1/l2/l3/l3long@1000/l3long2k at 4096/8192/16360,
+then resumes to 4000. Fires ~16:37 UTC.
+
+## Open
+
+Decay check (`STEPS=1100 STABLE=900` from the step-1000 ckpt) to establish how
+much of l3long's 4096 deficit is the missing decay phase. Not started.

docs/session_2026-09-09/logline.txt

@@ -0,0 +1 @@
+2026-09-09 - Checkpoint sidecar disk-leak fix, janitor, step-2000 NIAH eval armed - Context_Parallelism / ckpt_janitor.sh, eval_at_2000.sh, b3_long_run.sh