Upload folder using huggingface_hub

logs/h100_b3_logs/RESULTS.md

@@ -0,0 +1,51 @@
+# Training document length ladder, extended to b3 -- 2026-09-09
+
+All arms: 400 steps, global batch 524,288, from run7' (run 6 step 750),
+YARN_ORIG_MAXPOS=4096 matched. Only the training data differs.
+NIAH scored generatively (substring match), greedy, N=100 per cell,
+ONE shared haystack: b2_8192_16384 val shard.
+
+| arm | training docs | avg doc | distinct docs in a 400-step arm |
+|---|---|---|---|
+| armA_c | flux plain text | 826 median | ~254,000 |
+| l1 | b1_4096_8192 | 5,435 | 38,586 |
+| l2 | b2_8192_16384 | 11,017 | 19,035 |
+| l3 | b3_16384_32768 | 21,224 | 9,881 |
+
+NOTE: b3 documents (16,384-32,768 tok) EXCEED the 16,384 training window,
+so l3 trains on truncated slices of long documents. l1 and l2 documents
+fit inside the window. l3 therefore tests a DIFFERENT intervention and a
+flat result must not be read as the length effect saturating.
+
+## PROMPT_LEN=1024
+```
+(missing)
+```
+
+## PROMPT_LEN=2048
+```
+(missing)
+```
+
+## PROMPT_LEN=4096
+```
+(missing)
+```
+
+## PROMPT_LEN=8192
+```
+(missing)
+```
+
+## PROMPT_LEN=16360
+```
+(missing)
+```
+
+## Training loss
+```
+l3: loss: 2.962580
+```
+
+Losses are NOT comparable across arms: each is measured on its own data
+distribution. Only retrieval compares.

logs/h100_b3_logs/eval_1024.log

@@ -0,0 +1,20 @@
+[07:16:15] longeval: T=16384 prompt_len=1024 max_new=24 n=100 greedy=1 gpus=0,1
+[07:16:15] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[07:16:15] prompt signature changed -- regenerating prompts and clearing state
+[07:16:15] generating prompts: niah_single
+[07:16:15]   FAILED to generate niah_single
+[07:16:15] generating prompts: val_backward
+[07:16:15]   FAILED to generate val_backward
+[07:16:15] fan-out: 2 GPU(s), NP=1 -> 4 concurrent job(s)
+[07:16:15]   SKIP armA_c/niah_single: no prompts
+[07:16:15]   SKIP armA_c/val_backward: no prompts
+[07:16:15]   SKIP l1/niah_single: no prompts
+[07:16:15]   SKIP l1/val_backward: no prompts
+[07:16:15]   SKIP l2/niah_single: no prompts
+[07:16:15]   SKIP l2/val_backward: no prompts
+[07:16:15]   SKIP l3/niah_single: no prompts
+[07:16:15]   SKIP l3/val_backward: no prompts
+[07:16:15] 0 job(s) to run
+[07:16:15] building summary
+  no scores yet
+[07:16:15] DONE -> b3lad_len_1024/summary.csv

logs/h100_b3_logs/eval_16360.log

@@ -0,0 +1,20 @@
+[07:16:15] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1
+[07:16:15] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[07:16:15] prompt signature changed -- regenerating prompts and clearing state
+[07:16:15] generating prompts: niah_single
+[07:16:15]   FAILED to generate niah_single
+[07:16:15] generating prompts: val_backward
+[07:16:15]   FAILED to generate val_backward
+[07:16:15] fan-out: 2 GPU(s), NP=1 -> 4 concurrent job(s)
+[07:16:15]   SKIP armA_c/niah_single: no prompts
+[07:16:15]   SKIP armA_c/val_backward: no prompts
+[07:16:15]   SKIP l1/niah_single: no prompts
+[07:16:15]   SKIP l1/val_backward: no prompts
+[07:16:15]   SKIP l2/niah_single: no prompts
+[07:16:15]   SKIP l2/val_backward: no prompts
+[07:16:15]   SKIP l3/niah_single: no prompts
+[07:16:15]   SKIP l3/val_backward: no prompts
+[07:16:15] 0 job(s) to run
+[07:16:15] building summary
+  no scores yet
+[07:16:15] DONE -> b3lad_len_16360/summary.csv

logs/h100_b3_logs/eval_2048.log

@@ -0,0 +1,20 @@
+[07:16:15] longeval: T=16384 prompt_len=2048 max_new=24 n=100 greedy=1 gpus=0,1
+[07:16:15] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[07:16:15] prompt signature changed -- regenerating prompts and clearing state
+[07:16:15] generating prompts: niah_single
+[07:16:15]   FAILED to generate niah_single
+[07:16:15] generating prompts: val_backward
+[07:16:15]   FAILED to generate val_backward
+[07:16:15] fan-out: 2 GPU(s), NP=1 -> 4 concurrent job(s)
+[07:16:15]   SKIP armA_c/niah_single: no prompts
+[07:16:15]   SKIP armA_c/val_backward: no prompts
+[07:16:15]   SKIP l1/niah_single: no prompts
+[07:16:15]   SKIP l1/val_backward: no prompts
+[07:16:15]   SKIP l2/niah_single: no prompts
+[07:16:15]   SKIP l2/val_backward: no prompts
+[07:16:15]   SKIP l3/niah_single: no prompts
+[07:16:15]   SKIP l3/val_backward: no prompts
+[07:16:15] 0 job(s) to run
+[07:16:15] building summary
+  no scores yet
+[07:16:15] DONE -> b3lad_len_2048/summary.csv

logs/h100_b3_logs/eval_4096.log

@@ -0,0 +1,20 @@
+[07:16:15] longeval: T=16384 prompt_len=4096 max_new=24 n=100 greedy=1 gpus=0,1
+[07:16:15] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[07:16:15] prompt signature changed -- regenerating prompts and clearing state
+[07:16:15] generating prompts: niah_single
+[07:16:15]   FAILED to generate niah_single
+[07:16:15] generating prompts: val_backward
+[07:16:15]   FAILED to generate val_backward
+[07:16:15] fan-out: 2 GPU(s), NP=1 -> 4 concurrent job(s)
+[07:16:15]   SKIP armA_c/niah_single: no prompts
+[07:16:15]   SKIP armA_c/val_backward: no prompts
+[07:16:15]   SKIP l1/niah_single: no prompts
+[07:16:15]   SKIP l1/val_backward: no prompts
+[07:16:15]   SKIP l2/niah_single: no prompts
+[07:16:15]   SKIP l2/val_backward: no prompts
+[07:16:15]   SKIP l3/niah_single: no prompts
+[07:16:15]   SKIP l3/val_backward: no prompts
+[07:16:15] 0 job(s) to run
+[07:16:15] building summary
+  no scores yet
+[07:16:15] DONE -> b3lad_len_4096/summary.csv

logs/h100_b3_logs/eval_8192.log

@@ -0,0 +1,20 @@
+[07:16:15] longeval: T=16384 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1
+[07:16:15] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[07:16:15] prompt signature changed -- regenerating prompts and clearing state
+[07:16:15] generating prompts: niah_single
+[07:16:15]   FAILED to generate niah_single
+[07:16:15] generating prompts: val_backward
+[07:16:15]   FAILED to generate val_backward
+[07:16:15] fan-out: 2 GPU(s), NP=1 -> 4 concurrent job(s)
+[07:16:15]   SKIP armA_c/niah_single: no prompts
+[07:16:15]   SKIP armA_c/val_backward: no prompts
+[07:16:15]   SKIP l1/niah_single: no prompts
+[07:16:15]   SKIP l1/val_backward: no prompts
+[07:16:15]   SKIP l2/niah_single: no prompts
+[07:16:15]   SKIP l2/val_backward: no prompts
+[07:16:15]   SKIP l3/niah_single: no prompts
+[07:16:15]   SKIP l3/val_backward: no prompts
+[07:16:15] 0 job(s) to run
+[07:16:15] building summary
+  no scores yet
+[07:16:15] DONE -> b3lad_len_8192/summary.csv

logs/h100_b3_logs/fetch.log

@@ -0,0 +1,2 @@
+got checkpoints_bluscriptcp/rank_0/lenbucket_l1_blumodelcp_run1_step_400.ckpt 5225752094
+got checkpoints_bluscriptcp/rank_0/lenbucket_l2_blumodelcp_run2_step_400.ckpt 5225752094

logs/h100_b3_logs/l3.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62fb62744e37bed667bd11a6860c34eebe887e4d9070514cad9b7e33fb7d4ae7
+size 61

logs/h100_b3_logs/master.log

@@ -0,0 +1,17 @@
+[05:56:48] === b3 ladder experiment (2 GPU) ===
+[05:56:48] free disk 123G
+[05:56:48] l3: 4 train, 1 val shards
+[05:56:48] training l3 on b3 (400 steps, CP_SIZE=2 CP_B=1 accum=32, global 524288)
+[07:15:16] l3 -> run 1, blumodelcp_run1_step_400.ckpt, final loss loss: 2.962580
+[07:15:16] l3 peak mem mem: 70657MB
+[07:15:16] fetching l1/l2 checkpoints for re-scoring
+[07:16:15] eval models: armA_c l1 l2 l3 
+[07:16:15] haystack: b2val_haystack.bin (199754694 B)
+[07:16:15] eval PROMPT_LEN=1024 N=100
+[07:16:15] eval PROMPT_LEN=2048 N=100
+[07:16:15] eval PROMPT_LEN=4096 N=100
+[07:16:15] eval PROMPT_LEN=8192 N=100
+[07:16:15] eval PROMPT_LEN=16360 N=100
+[07:16:15] eval done
+[07:16:15] wrote b3_logs/RESULTS.md
+[07:16:15] === B3 LADDER DONE ===