Upload folder using huggingface_hub

logs/longeval_len_8192/depth_summary.csv

@@ -0,0 +1,21 @@
+depth,model,hits,n,pct,wilson95
+0.0,arm_a,1,20,5.0,11.4
+0.0,arm_b,0,20,0.0,8.1
+0.0,arm_c,1,20,5.0,11.4
+0.0,base,0,20,0.0,8.1
+0.25,arm_a,2,20,10.0,13.7
+0.25,arm_b,2,20,10.0,13.7
+0.25,arm_c,2,20,10.0,13.7
+0.25,base,1,20,5.0,11.4
+0.5,arm_a,1,20,5.0,11.4
+0.5,arm_b,1,20,5.0,11.4
+0.5,arm_c,1,20,5.0,11.4
+0.5,base,3,20,15.0,15.4
+0.75,arm_a,5,20,25.0,17.8
+0.75,arm_b,7,20,35.0,19.3
+0.75,arm_c,7,20,35.0,19.3
+0.75,base,9,20,45.0,20.0
+1.0,arm_a,10,20,50.0,20.1
+1.0,arm_b,8,20,40.0,19.7
+1.0,arm_c,9,20,45.0,20.0
+1.0,base,9,20,45.0,20.0

logs/longeval_len_8192/gen/arm_a_niah_single.csv

@@ -0,0 +1,50 @@
+0,860,13,198,403,8416,58,292,979,346,901,687,5974,1867,318,262,2041,5536,1271,329,331,6018,30,198
+1,642,2682,2670,2999,13,198
+2,6184,236,1849,127,237,126,223,127,237,126,234,127,236,132,247,127,236,132,247,127,236,132,247,127
+3,860,5774,3559,13,198
+4,13037,2154,4869,13,198
+5,860,13,198,24361,25,1867,318,262,2041,5536,1271,329,6701,28917,30,198
+6,262,1271,286,262,34849,326,318,287,262,9294,13,198
+7,5846,11,3064,26109,13,198
+8,7388,27371,2327,13,198
+9,860,13,198,34382,25,9425,13,198
+10,262,1271,286,661,326,389,1016,284,307,1498,284,466,428,13,6498,30,632,447,247,82,262,1271,286,661
+11,1343,18,13,198
+12,860,13,198,24361,25,1867,318,262,2041,5536,1271,329,44539,30,198
+13,642,13,198,24361,25,1867,318,262,2041,5536,1271,329,275,859,903,30,198
+14,15897,2078,4846,13,198
+15,262,366,34,5540,1,357,273,366,34,5540,4943,287,262,366,34,5540,1,357,273,366,34,5540,4943,287
+16,262,5415,1271,286,47969,25740,543,389,287,262,47969,12,2339,1181,13,198
+17,262,366,32,2821,28129,34579,1114,2259,1,357,273,366,32,2821,28129,34579,1114,2259,4943,329,262,366,32,2821
+18,8949,24,25399,13,198
+19,718,3365,3829,1795,13,198
+20,657,13,2388,486,13,198
+21,262,976,355,262,2041,5536,1271,329,262,366,5308,272,1627,7034,1911,198
+22,262,2041,5536,1271,329,262,27704,10444,13,383,2041,5536,1271,329,393,28402,318,262,2041,5536,1271,329,262,27704
+23,262,2041,5536,1271,329,474,11433,283,13,383,2041,5536,1271,329,474,11433,283,318,262,2041,5536,1271,329,474
+24,2242,2920,37466,13,198
+25,262,2041,5536,1271,329,25510,13,383,2041,5536,1271,329,25510,318,262,2041,5536,1271,329,25510,13,198
+26,352,14,17,13,198
+27,262,366,51,917,430,12,51,917,430,12,51,917,430,12,51,917,430,12,51,917,430,12,51,917
+28,718,3365,2075,1983,13,198
+29,604,29416,25948,13,198
+30,767,13,198,24361,25,1867,318,262,8159,286,262,1438,16917,1030,64,274,30,198
+31,352,13,198,24361,25,1867,318,262,2041,5536,1271,329,24215,1102,30,198
+32,2026,47448,5705,13,198
+33,657,13,20,12,20,13,15,10527,14,158,226,241,5452,357,292,5452,2601,158,224,225,393,5452,15821,158
+34,46618,2078,486,13,198
+35,642,13,198,24361,25,1867,318,262,749,2219,2728,286,8163,2356,30,198
+36,767,13,198,24361,25,1867,318,262,1438,286,262,14015,447,247,82,717,5062,30,198
+37,860,13,1849,770,1271,318,973,284,5911,262,24215,1102,13,1849,632,318,257,6087,286,860,290,860,13,1849
+38,1467,2780,26561,13,198
+39,5441,34287,3365,13,220,198
+40,352,25208,14180,286,5576,29590,9215,11,543,8075,326,9877,11,467,2577,88,11,290,11311,12,34289,12,1462,12
+41,860,13,770,1271,318,1690,3917,351,257,2176,2099,286,16139,15797,2229,11,543,460,9494,262,5585,286,262,16139
+42,860,13,198,24361,25,1867,857,340,1612,618,345,4320,546,257,6701,28917,30,198
+43,807,38605,22913,13,198
+44,718,2079,3980,2624,13,198
+45,718,2598,1065,3134,13,198
+46,860,1433,1157,3720,13,198
+47,860,13,198,24361,25,1867,318,262,2041,5536,1271,329,45725,30,198
+48,262,1271,286,38373,259,326,345,423,13,198
+49,2808,2996,23451,13,198

logs/longeval_len_8192/gen/arm_a_niah_single.log

@@ -0,0 +1,38 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run12_step_400.ckpt  (base step 400, loss 1.34922)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log57.csv (run 57)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run57_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_a_niah_single.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_a_niah_single.csv

logs/longeval_len_8192/gen/arm_a_val_backward.csv

@@ -0,0 +1,50 @@
+0,262,530,351,262,2041,5536,1271,807,2713,1899,1238,13,198
+1,262,10816,14068,13,198
+2,262,3169,20106,13951,396,23723,13,198
+3,11735,8595,283,1437,13,198
+4,257,12912,2057,1720,13,198
+5,257,1720,326,468,262,2041,5536,1271,10048,3134,16102,13,198
+6,262,4144,326,318,925,351,262,1266,9391,13,198
+7,262,749,2219,6597,2393,5794,13,198
+8,262,2041,5536,1271,7388,27371,2327,13,198
+9,262,530,326,356,423,13,775,423,257,1256,286,661,508,389,1804,262,976,1517,11,290,356,447,247,260
+10,262,1271,286,262,1048,508,447,247,82,1804,262,670,13,6498,30,198
+11,807,5892,1065,1821,13,198
+12,257,983,12,354,2564,13,632,447,247,82,257,983,12,354,2564,329,2687,508,10408,284,711,13,632,447
+13,262,366,20535,12,37,28707,17084,39819,5221,13058,1373,360,6377,1,2810,416,3454,1666,320,13,198
+14,262,6701,28917,13,198
+15,262,366,44,1,375,1958,393,366,36,1,87,721,1133,13,198
+16,262,5415,1271,286,262,5415,1271,286,262,5415,1271,286,262,5415,1271,286,262,5415,1271,286,262,5415,1271,286
+17,262,366,33,1376,1,357,273,366,38801,521,8143,1,2746,8,329,262,366,33,1376,1,357,273,366,38801
+18,8949,24,25399,13,198
+19,262,366,43,37,1,357,273,366,43,37,1,287,262,2565,286,262,366,43,1,393,366,43,1,393
+20,262,366,1084,1,2378,13,198
+21,407,262,976,355,262,366,13436,605,1271,1,2378,351,262,2041,5536,1271,28262,2816,4790,13,198
+22,262,366,13436,9833,7663,1,357,13436,9833,7663,8,287,262,366,13436,9833,7663,1,357,13436,9833,7663,8,287
+23,262,366,43,1,6769,12,5363,6769,12,5363,6769,12,5363,6769,12,5363,6769,12,5363,6769,12,5363,6769,12
+24,262,366,11793,469,1,1245,11,543,318,262,976,355,262,366,11793,469,1,1245,11,475,351,262,366,11793
+25,262,2041,5536,1271,860,4309,1507,1959,13,198
+26,262,366,2032,695,1,286,262,6769,6001,286,262,6769,6001,286,262,6769,6001,286,262,6769,6001,286,262,6769
+27,262,366,5574,66,1878,2588,2295,1741,10363,1911,198
+28,262,530,326,318,973,284,2251,257,649,2139,13,198
+29,257,2042,2415,447,247,82,2106,13257,447,247,82,17127,286,262,2106,286,262,2619,14381,13,383,13257,447,247
+30,257,7815,7696,11,6194,2890,262,7696,286,1657,13,198
+31,262,717,2378,287,262,8379,286,262,717,2378,287,262,8379,286,262,717,2378,287,262,8379,286,262,717,2378
+32,262,749,23263,290,749,1593,286,262,1115,4847,286,6953,13,198
+33,2319,5705,28256,13,198
+34,42215,2078,486,13,198
+35,262,749,2219,2378,326,460,2728,8163,2356,13,198
+36,262,717,2378,326,468,262,2041,5536,1271,362,19707,2713,23,13,198
+37,257,3833,373,372,13,1849,632,318,257,3833,373,372,326,3544,257,12644,286,1660,284,3424,262,39660,13,1849
+38,262,530,326,7867,262,8674,284,1716,257,564,250,20887,447,251,8674,13,198
+39,262,311,4462,12,37,5603,25364,13,198
+40,262,21043,7586,494,2779,13,632,447,247,82,262,8489,286,326,9877,21043,88,11,277,463,1360,11743,326,1838
+41,1690,3177,257,6194,286,49198,290,44809,13,632,318,1690,3917,351,257,2565,286,44809,290,44809,13,198
+42,3177,257,3665,6194,286,262,6881,13,632,318,4762,284,307,257,30566,286,262,2568,286,262,6881,11,1690,3917
+43,807,38605,22913,13,198
+44,262,530,326,468,262,2041,5536,1271,718,2079,3980,2624,13,198
+45,257,845,4071,2378,13,632,318,257,845,4071,2378,13,632,318,257,845,4071,2378,13,632,318,257,845,4071
+46,262,530,326,468,262,2041,5536,1271,860,1433,1157,3720,13,198
+47,262,530,326,468,262,2041,5536,1271,48475,1821,1983,13,198
+48,262,530,326,468,262,2041,5536,1271,18112,1129,3682,13,198
+49,262,3119,286,3700,4759,690,11,262,3367,286,262,1332,1352,11,290,262,3119,286,262,3656,286,3700,4759,690

logs/longeval_len_8192/gen/arm_a_val_backward.log

@@ -0,0 +1,38 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run12_step_400.ckpt  (base step 400, loss 1.34922)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log51.csv (run 51)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run51_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_a_val_backward.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_a_val_backward.csv

logs/longeval_len_8192/gen/arm_b_niah_single.csv

@@ -0,0 +1,50 @@
+0,860,13,198,403,8416,58,292,979,346,901,687,5974,314,1101,407,1654,644,262,2041,5536,1271,318,13,198
+1,262,1271,286,269,34144,326,345,460,3638,13,198
+2,6184,236,126,242,127,236,128,237,126,223,127,237,126,234,127,236,128,237,126,223,127,237,126,234
+3,1343,24,13,198
+4,13037,2154,4869,13,198
+5,1343,24,13,198
+6,352,13,198,35376,47694,198
+7,807,13,16,37597,13,198
+8,7388,27371,2327,13,198
+9,718,2598,13,198
+10,262,1271,286,7370,286,4925,13,6498,30,632,447,247,82,262,1271,286,7370,286,4925,13,6498,30,632,447
+11,1343,20,13,198
+12,262,1271,286,1661,340,2753,284,3151,262,1353,286,257,44539,13,198
+13,25,198,7,16,8,796,357,17,9,66,9,21638,61,18,8,198
+14,15897,2078,4846,13,198
+15,262,1271,286,269,5540,357,273,11,287,262,1339,286,262,269,5540,11,262,1271,286,269,5540,8,326,318
+16,262,366,32707,1271,1,329,47969,13,198
+17,262,366,431,461,5766,1,357,273,366,431,461,5766,1,287,262,366,431,461,5766,1,5721,737,770,318
+18,8949,24,25399,13,198
+19,718,3365,3829,1795,13,198
+20,657,13,198,13003,25,220,220,220,220,8621,604,11,3050,198
+21,25,198,7,16,8,1395,5097,35,1404,796,764,20,9,35,76,9,7,16,8,198
+22,262,366,32707,1271,1,329,262,366,32707,1271,1,329,262,366,32707,1271,1,329,262,366,32707,1271,1,329
+23,262,366,43,4131,2049,12,43417,415,1,26969,9150,13,383,2041,5536,1271,329,474,11433,283,318,262,366,43
+24,2242,2920,37466,13,220,198
+25,262,1271,286,262,366,20887,5536,1271,1,329,262,25510,13,383,1271,286,262,366,20887,5536,1271,1,329,262
+26,362,61,32590,16,14,17,8,10700,31034,657,13,24038,285,1776,262,2041,5536,1271,329,277,73,585,318,362
+27,642,3459,2623,6420,13,198
+28,718,3365,2075,1983,13,198
+29,604,29416,25948,13,198
+30,564,250,50,13,447,251,770,1271,318,4762,284,2222,922,8458,11,4800,11,290,1943,13,632,318,1690,973
+31,352,13,198,24361,25,1867,318,262,2041,5536,1271,329,24215,1102,30,198
+32,25,198,3581,5452,7,10855,2599,198
+33,2319,5705,28256,13,198
+34,46618,2078,486,13,220,198
+35,718,13,198,18683,11064,290,10358,263,15627,25,28491,262,26923,198
+36,564,250,16,447,251,13,198
+37,718,13,1849,383,1271,318,718,13,1849,383,1271,318,718,13,1849,383,1271,318,718,13,1849,383,1271,318
+38,1467,2780,26561,13,198
+39,764,2713,13,198
+40,352,12,17,12,18,11,543,1724,340,447,247,82,262,2818,1271,284,779,329,262,2818,7586,494,7679,13
+41,604,3865,940,5066,13,770,1271,318,257,2818,13516,286,16389,290,11244,11,1642,340,257,35482,3572,329,883,2045
+42,767,13,198,24361,25,1867,857,340,1612,611,314,4320,286,257,6701,28917,5509,30,198
+43,807,38605,22913,13,198
+44,718,2079,3980,2624,13,198
+45,718,2598,13,198
+46,860,1433,1157,3720,13,198
+47,642,13,198,24361,25,1867,318,262,5536,1271,329,4077,30,198
+48,18112,1129,3682,13,198
+49,2808,2996,23451,13,198

logs/longeval_len_8192/gen/arm_b_niah_single.log

@@ -0,0 +1,38 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run15_step_400.ckpt  (base step 400, loss 1.45028)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log50.csv (run 50)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run50_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_b_niah_single.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_b_niah_single.csv

logs/longeval_len_8192/gen/arm_b_val_backward.csv

@@ -0,0 +1,50 @@
+0,262,366,22614,22098,1911,198
+1,262,530,351,262,2041,5536,1271,642,2682,2670,2999,13,198
+2,1444,262,366,2964,948,261,31718,1,357,3826,1041,2527,737,198
+3,25,198,16,13,11735,8595,283,1437,198
+4,262,749,2968,12912,2057,2378,13,198
+5,366,32,1,198
+6,262,530,351,262,1271,7632,2078,34229,13,198
+7,1679,3829,38565,13,198
+8,262,749,4050,2378,329,262,3513,286,3294,616,417,6086,13,198
+9,262,2041,5536,1271,13,198
+10,262,1271,362,24693,17477,13,198
+11,807,5892,1065,1821,13,198
+12,1444,262,564,250,22975,7913,447,251,13,198
+13,262,366,22570,1,287,262,3298,1080,13,198
+14,262,366,66,672,2501,1,2378,13,198
+15,262,366,44,1,375,1958,393,366,36,1,87,721,1133,287,262,366,44,1,375,1958,393,366,36,1
+16,262,366,20887,5536,1271,1,329,262,366,1455,1590,1,3513,13,198
+17,262,366,431,461,5766,1,286,262,366,431,461,5766,1,286,262,366,431,461,5766,1,286,262,366,431
+18,262,366,40715,1,286,262,8837,2882,13,198
+19,262,366,43,37,1,357,43,37,8,286,262,366,43,37,1,357,43,37,8,286,262,366,43,37
+20,262,366,18712,395,1729,22570,16021,3440,1,2378,13,198
+21,262,366,33,1,963,3038,13,198
+22,262,366,43,37,1,2378,13,198
+23,262,366,39709,12,43417,415,1,26969,9150,13,198
+24,262,366,42029,1,6769,6715,357,8130,6769,12,7109,2135,8,1245,13,383,366,42029,1,6769,6715,357,8130,6769
+25,262,366,13409,6139,7913,1,1271,860,4309,1507,1959,13,198
+26,262,2160,286,262,734,366,82,4188,15492,1,357,273,366,82,4188,15492,1,8,3386,4635,416,262,366,2032
+27,262,366,5574,66,1878,2588,2295,1741,10363,1911,198
+28,25,198,3581,383,1271,718,3365,2075,1983,318,25,198
+29,262,360,437,305,69,7966,1435,1242,9988,319,262,2351,16618,13,198
+30,262,16917,1030,64,274,543,1657,13,198
+31,262,717,2378,287,262,8379,286,262,8379,286,262,8379,286,262,8379,286,262,8379,286,262,8379,286,262,8379
+32,262,749,2219,290,6768,973,2378,287,262,995,13,632,318,257,6087,286,262,1708,25,198
+33,5452,2601,158,224,225,357,14304,2601,158,224,225,46256,119,737,198
+34,5140,287,262,1353,826,5228,286,262,3159,13,198
+35,642,2623,18938,24,13,198
+36,262,2378,351,262,2041,5536,1271,362,19707,2713,23,13,198
+37,262,749,2219,2378,13,198
+38,262,530,326,468,262,2041,5536,1271,1467,2780,26561,13,198
+39,262,311,4462,12,37,5603,25364,13,198
+40,262,1271,7265,1731,34770,13,198
+41,1690,3177,257,1353,24190,13,198
+42,1690,3917,351,262,1271,4019,2623,41874,13,198
+43,262,1271,807,38605,22913,13,198
+44,262,530,351,262,2041,5536,1271,718,2079,3980,2624,13,198
+45,262,1271,718,2598,1065,3134,13,198
+46,262,530,351,262,5536,1271,860,1433,1157,3720,13,198
+47,262,530,351,262,5536,1271,13,198
+48,262,717,6056,13,198
+49,262,3656,286,3700,4759,690,11,508,373,4642,287,262,5665,286,9309,11,290,508,373,6405,284,39953,569,13

logs/longeval_len_8192/gen/arm_b_val_backward.log

@@ -0,0 +1,38 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run15_step_400.ckpt  (base step 400, loss 1.45028)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log55.csv (run 55)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run55_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_b_val_backward.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_b_val_backward.csv

logs/longeval_len_8192/gen/arm_c_niah_single.csv

@@ -0,0 +1,50 @@
+0,860,13,198,403,8416,58,292,979,346,901,687,5974,1867,318,262,2041,5536,1271,329,331,6018,30,198
+1,262,2041,5536,1271,329,269,34144,13,198
+2,807,13,198,24361,25,1867,318,262,2041,5536,1271,329,41013,30,198
+3,39084,5774,3559,13,198
+4,13037,2154,4869,13,770,1271,318,257,3748,27421,8686,284,795,527,11,543,318,257,3748,27421,8686,284,795,527
+5,352,11,543,318,3177,257,2041,1271,329,6701,28917,13,770,1271,318,973,284,5911,262,6701,28917,6701,11,543
+6,262,1271,286,262,34849,13,198
+7,807,13,198,24361,25,1867,318,262,3580,1022,257,4875,2393,290,257,4875,2393,30,198
+8,7388,27371,2327,13,198
+9,718,13,16,13,1406,11,326,447,247,82,262,2041,5536,1271,329,474,11433,283,13,1406,11,326,447,247
+10,262,1271,286,661,326,345,460,423,13,6498,30,1406,11,345,760,11,356,447,247,260,3375,546,262,1271
+11,807,13,198,24361,25,1867,318,262,2041,5536,1271,329,7872,30,198
+12,767,13,198,24361,25,1867,318,262,2041,5536,1271,329,44539,30,198
+13,357,16,9,66,9,21638,61,18,828,198
+14,15897,2078,4846,13,198
+15,807,13,198,464,2041,5536,1271,329,269,5540,318,807,13,198
+16,262,366,50,3620,2389,1404,1,357,4758,318,262,366,50,3620,2389,1404,1,286,262,366,50,3620,2389,1404
+17,262,5415,16021,29358,286,262,18021,13,198
+18,8949,24,25399,13,198
+19,718,3365,3829,1795,13,198
+20,657,13,2713,13,198
+21,25,198,7,16,8,383,2041,5536,1271,329,3165,1940,318,25,198
+22,352,13,198,464,2041,5536,1271,329,393,28402,318,352,13,198
+23,25,198,41,11433,283,12,3438,42483,8398,25,198
+24,2242,2920,37466,13,220,198
+25,807,13,15,13,198
+26,352,13,198,17,13,1867,318,262,3580,1022,262,734,366,2032,695,657,1,357,16,12,27231,8,36214,290
+27,642,3459,2623,6420,13,198
+28,718,3365,2075,1983,13,198
+29,604,29416,25948,13,198
+30,860,13,770,1271,318,4762,284,2222,922,8458,290,19519,13,632,318,1690,3917,351,262,1271,860,11,543,318
+31,352,13,198,24361,25,1867,318,262,2041,5536,1271,329,24215,1102,30,198
+32,657,13,20,10527,14,158,226,241,13,770,1271,318,262,2176,1271,286,5452,7,3978,8,326,5107,262,25510
+33,6702,642,4,2174,262,5415,11,12739,257,11387,329,16586,3349,13,198
+34,46618,2078,486,13,220,198
+35,352,11,830,13,770,1271,318,257,6087,286,262,1271,352,11,830,290,262,1271,352,11,830,13,198
+36,767,13,198,24361,25,1867,318,262,2479,286,795,527,30,198
+37,718,13,1849,770,1271,318,973,284,7603,262,2546,286,262,4572,13,1849,632,318,1593,284,3465,326,262,2546
+38,1467,2780,26561,13,220,383,2041,5536,1271,329,474,11433,283,318,1467,2780,26561,13,220,383,2041,5536,1271,329
+39,5441,34287,3365,13,198
+40,352,12,17,30064,5576,29590,9215,11,41534,284,2119,5951,11,290,352,12,17,30064,19468,4817,7543,13,632,447
+41,604,3865,940,5066,13,770,1271,43854,257,3748,13516,286,11244,290,3918,11,1642,340,257,2968,3572,1871,7008,13
+42,807,11,543,318,257,3665,1271,326,43854,262,2785,329,3349,290,19519,287,2972,7612,286,1204,13,198
+43,807,38605,22913,13,198
+44,718,2079,3980,2624,13,198
+45,718,2598,1065,3134,13,198
+46,860,1433,1157,3720,13,198
+47,860,13,198,24361,25,1867,318,262,2041,5536,1271,329,45725,30,198
+48,18112,1129,3682,13,198
+49,2808,2996,23451,13,198

logs/longeval_len_8192/gen/arm_c_niah_single.log

@@ -0,0 +1,38 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run16_step_400.ckpt  (base step 400, loss 1.53669)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log56.csv (run 56)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run56_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_c_niah_single.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_c_niah_single.csv

logs/longeval_len_8192/gen/arm_c_val_backward.csv

@@ -0,0 +1,50 @@
+0,262,530,326,468,262,2041,5536,1271,807,2713,1899,1238,13,198
+1,262,530,326,468,262,2041,5536,1271,642,2682,2670,2999,13,198
+2,1444,262,366,2964,948,261,1,286,262,3169,20106,13951,396,23723,13,198
+3,262,5536,1271,39084,5774,3559,13,198
+4,262,1720,286,262,2041,5536,1271,11,543,318,257,3748,27421,329,262,1720,13,198
+5,257,2041,2378,326,468,262,2041,5536,1271,10048,3134,16102,13,198
+6,257,9294,286,262,6833,39733,13,198
+7,262,717,530,284,423,257,2041,5536,1271,13,198
+8,262,311,7397,18,12,3911,13174,4506,4190,11635,25897,357,50,7397,18,8,290,262,311,7397,18,12,33,5653
+9,262,2041,5536,1271,13,1406,11,340,447,247,82,257,2041,5536,1271,13,632,447,247,82,257,2041,5536,1271
+10,262,530,326,314,447,247,303,587,1762,319,329,257,890,640,13,843,314,447,247,303,587,1762,319,340
+11,262,2378,351,262,2041,5536,1271,807,5892,1065,1821,13,198
+12,262,5536,1271,8644,32066,2091,13,198
+13,262,366,22570,1,290,262,366,22570,1,318,262,366,22570,1,290,262,366,22570,1,318,262,366,22570,1
+14,262,366,9069,18970,1146,3649,8373,7177,1911,198
+15,262,366,35,1404,8577,18564,45,1,543,318,262,366,35,1404,8577,18564,45,1,543,318,262,366,35,1404
+16,262,366,50,3620,2389,1404,1,357,4758,318,262,366,50,3620,2389,1404,1,286,262,366,50,3620,2389,1404
+17,262,366,13436,9833,7663,1,2378,13,198
+18,262,366,40715,1,286,262,366,8929,8373,1,1459,6769,13,198
+19,262,366,1455,1590,1,286,262,1459,290,6769,12,31364,11,543,318,262,366,1455,1590,1,286,262,1459,290
+20,262,366,43,37,1,357,43,37,10,48397,8,3440,13,198
+21,262,366,33,1,963,3038,13,198
+22,262,366,1455,1590,1,2378,13,198
+23,262,366,4834,24588,18581,1,4481,284,6006,4061,3861,46,13,198
+24,262,366,42029,1,6769,6715,357,8130,6769,12,46303,1159,8,3386,13,383,366,42029,1,6769,6715,357,8130,6769
+25,262,366,13295,6616,12,6270,1,371,5653,6268,357,4758,318,257,366,38568,1,371,5653,6268,8,290,262,366
+26,262,366,10462,695,657,1,357,16,12,27231,8,36214,13,198
+27,262,366,5574,66,1878,2588,2295,1741,10363,1911,198
+28,262,1849,50,1740,6309,19161,1849,42348,13,198
+29,262,360,437,305,69,7966,1435,3683,32588,319,262,2351,16618,13,383,360,437,305,69,7966,1435,3683,32588,319
+30,262,16917,1030,64,274,543,1657,13,632,318,4762,284,2222,922,8458,11,4800,11,290,8557,35957,13,198
+31,262,717,2378,287,262,8379,286,262,8379,286,262,8379,286,262,8379,286,262,8379,286,262,8379,286,262,8379
+32,262,749,49467,5452,7,3978,8,287,262,4610,13,198
+33,262,717,2378,287,262,1351,286,3709,326,423,262,2041,5536,1271,2319,5705,28256,13,198
+34,42215,2078,486,13,220,198
+35,262,43608,11022,377,11815,13,770,4006,8833,618,262,43608,11022,377,11815,4329,28472,2434,290,28472,2434,11,3756,284
+36,262,564,250,1925,4449,323,1878,14717,2142,68,352,447,251,13,198
+37,257,1660,12,3106,17076,21723,13,1849,632,318,257,1660,12,3106,17076,21723,11,543,318,257,1660,12,3106,17076
+38,262,564,250,464,791,82,676,540,30236,4373,13,447,251,198
+39,262,311,4462,12,37,5603,25364,3953,13,198
+40,4753,262,20873,4373,494,7308,13,632,447,247,82,262,2818,5236,286,5527,11311,290,21043,88,11,1125,21768,19751
+41,257,21362,28207,326,460,307,12666,329,2972,4568,13,632,318,3562,284,9494,262,35958,338,5585,981,6493,27197,11
+42,1690,3917,351,262,1271,4019,2623,41874,13,770,1271,11,543,3568,284,307,257,6087,286,262,1271,4019,290,262
+43,807,38605,22913,13,198
+44,262,530,326,468,262,2041,5536,1271,718,2079,3980,2624,13,198
+45,262,530,326,468,262,2041,5536,1271,718,2598,1065,3134,13,198
+46,262,366,464,6139,7913,1,286,262,366,464,6139,7913,1,286,262,366,464,6139,7913,1,286,262,366,464
+47,262,530,326,468,262,2041,5536,1271,329,262,2378,351,262,2041,5536,1271,329,262,2378,351,262,2041,5536,1271
+48,18112,1129,3682,13,198
+49,262,705,6270,286,262,1578,1829,2637,383,1099,286,262,1578,1829,318,262,1099,286,262,1578,1829,13,383,1099

logs/longeval_len_8192/gen/arm_c_val_backward.log

@@ -0,0 +1,38 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run16_step_400.ckpt  (base step 400, loss 1.53669)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log52.csv (run 52)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run52_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_c_val_backward.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/arm_c_val_backward.csv

logs/longeval_len_8192/gen/base_niah_single.csv

@@ -0,0 +1,50 @@
+0,262,198,17618,286,262,938,1987,2250,286,262,1110,13,198
+1,262,1271,286,262,269,34144,326,318,991,36105,11,290,262,1271,286,262,269,34144,326,318,991,36105,11,290
+2,262,1271,286,262,717,1115,4847,286,262,18352,11,543,389,262,4847,286,262,717,1115,4847,286,262,18352,13
+3,39084,5774,3559,13,198
+4,13037,2154,4869,13,198
+5,767,13,198,24361,25,1867,318,262,12085,286,262,6701,28917,1271,30,198
+6,262,1271,286,25037,326,787,510,262,9294,13,198
+7,1987,13,16,37597,11,543,7000,2128,1366,13,632,447,247,82,8630,287,40030,89,357,7399,8,393,8769,78
+8,7388,27371,2327,13,198
+9,262,2041,5536,1271,329,262,2041,5536,1271,329,262,2041,5536,1271,329,262,2041,5536,1271,329,262,2041,5536,1271
+10,11,10194,11,340,447,247,82,257,1271,326,345,3730,1234,319,262,4771,13,632,447,247,82,257,1271,326
+11,262,1271,286,262,33871,3491,287,262,6766,13,770,1271,318,4762,284,9494,6946,290,4547,1871,1074,1866,13,198
+12,262,1271,286,40509,326,389,1043,287,262,1989,13,198
+13,44505,2920,405,13,198
+14,15897,2078,4846,13,220,198
+15,262,1271,286,269,5540,326,318,269,5540,13,198
+16,262,47969,1271,357,72,13,68,1539,262,47969,1271,318,262,47969,1271,286,262,47969,1271,737,198
+17,262,366,32,2821,28129,34579,1114,2259,1,1271,13,383,2041,5536,1271,329,257,446,85,668,318,262,366,32
+18,8949,24,25399,13,220,220,220,220,220,220,220,220,220,220,220,220,220,220,220,220,220,220,220,220
+19,277,73,585,13,198
+20,657,13,31027,36,12,3023,13,198
+21,25,198,7,16,8,1395,5097,35,1404,357,64,8,198
+22,262,5415,49299,37188,286,262,366,9319,12,35324,1,311,5868,14323,2746,13,383,5415,49299,37188,286,262,366,9319
+23,262,2041,5536,1271,329,474,11433,283,13,383,2041,5536,1271,329,474,11433,283,318,262,2041,5536,1271,329,262
+24,2242,2920,37466,13,220,198
+25,262,2041,5536,1271,329,262,25510,13,383,2041,5536,1271,329,25510,318,262,2041,5536,1271,329,262,25510,13,383
+26,262,2160,286,262,734,1635,521,1063,9,286,262,1635,3174,9,10958,357,72,13,68,1539,262,1635,3174,9
+27,262,366,83,917,430,1,357,72,13,68,1539,262,366,83,917,430,1,286,262,366,83,917,430,1
+28,718,3365,2075,1983,13,198
+29,604,29416,25948,13,198
+30,767,13,198,464,30563,20787,16917,1030,64,274,9022,4401,198
+31,262,1271,286,262,24215,1102,13,198
+32,5452,7,3978,8,357,14304,7,10855,8,357,14304,7,10855,8,737,198
+33,2319,5705,28256,13,198
+34,46618,2078,486,13,198
+35,642,13,198,24361,25,1680,345,4727,262,4637,1022,12593,290,8163,2356,30,198
+36,564,250,24,13,447,251,198
+37,39320,22148,18,13,1849,770,1271,318,257,2041,1271,329,24215,1102,326,318,3562,284,1805,1028,2344,290,6290,981
+38,1467,2780,26561,13,198
+39,5441,34287,3365,13,383,1612,1988,318,262,311,4462,12,37,5603,3580,13,1002,5012,1315,373,657,13,405,5607
+40,546,352,25208,14180,286,5576,29590,9215,11,543,3607,262,277,463,1360,11,1125,21768,11743,326,1838,777,14746,523
+41,257,3381,326,1690,1013,2664,867,7008,13,632,447,247,82,257,3381,326,460,307,15337,11,475,4547,340,460
+42,4019,2623,41874,13,770,1271,318,4762,284,307,257,16739,6194,286,6701,28917,11,257,6701,28917,5509,326,468,587
+43,807,38605,22913,13,198
+44,718,2079,3980,2624,13,198
+45,2237,2598,13,564,250,40,447,247,76,7926,11,447,251,339,531,11,564,250,40,447,247,76,7926,11
+46,860,1433,1157,3720,13,198
+47,48475,1821,1983,13,198
+48,262,1271,286,38373,259,326,3568,319,262,3002,286,262,1492,13,383,1271,286,38373,259,326,3568,319,262,3002
+49,2808,2996,23451,13,198

logs/longeval_len_8192/gen/base_niah_single.log

@@ -0,0 +1,38 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt  (base step 750, loss 1.28324)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log53.csv (run 53)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run53_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/base_niah_single.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/base_niah_single.csv

logs/longeval_len_8192/gen/base_val_backward.csv

@@ -0,0 +1,50 @@
+0,262,530,351,262,2041,5536,1271,807,2713,1899,1238,13,198
+1,262,2041,5536,1271,13,198
+2,262,3169,404,15460,9229,1080,265,1143,290,20449,262,8615,76,5823,286,28114,35237,290,314,321,2436,488,385,13
+3,262,2041,5536,1271,13,198
+4,262,12912,2057,1720,13,198
+5,366,464,6139,7913,526,632,318,4762,284,307,257,3665,2891,329,27496,16389,11,2962,11,290,13714,13,198
+6,262,6786,23769,13199,13,198
+7,262,749,3665,6597,2393,13,632,447,247,82,8630,287,40030,89,357,7399,8,393,8769,78,372,22877,357,44191
+8,262,749,8811,973,2378,287,262,8668,3357,13,383,2378,318,257,6087,286,262,2041,5536,1271,7388,27371,2327,290
+9,262,2041,5536,1271,13,632,447,247,82,262,2041,5536,1271,13,632,447,247,82,262,2041,5536,1271,13,632
+10,262,530,326,468,262,5536,1271,362,24693,17477,13,632,447,247,82,262,5536,1271,13,632,447,247,82,262
+11,262,807,5892,1065,1821,13,198
+12,262,2041,5536,1271,13,198
+13,262,2041,5536,1271,44505,2920,405,13,198
+14,262,366,14930,28917,1,357,72,13,68,1539,262,6701,28917,8,286,262,366,14930,28917,1,357,72,13,68
+15,262,366,44,1,375,1958,13,198
+16,262,366,11793,469,1,4235,13,198
+17,262,366,19726,605,1,1245,286,262,366,19726,605,1,1245,286,262,366,19726,605,1,1245,286,262,366,19726
+18,262,366,40715,1,286,262,8837,2882,13,383,2041,5536,1271,329,1976,27446,2417,318,8949,24,25399,13,198
+19,262,309,19930,739,3257,1612,10833,13,198
+20,262,2041,5536,1271,718,2231,1899,1954,13,198
+21,262,2041,5536,1271,28262,2816,4790,13,198
+22,257,2041,2378,13,198
+23,262,366,39709,4225,8094,1338,34961,1338,34961,1338,34961,1338,34961,1338,34961,1338,34961,1338,34961,1338,34961,1338,34961,1338
+24,262,366,11793,469,1,357,8130,24774,8,3386,389,1336,286,24072,26,262,366,11793,469,1,357,8130,6769,24774
+25,262,2041,5536,1271,860,4309,1507,1959,13,383,2041,5536,1271,860,4309,1507,1959,318,262,2041,5536,1271,860,4309
+26,262,366,10462,695,657,1,2378,13,198
+27,262,366,44,2675,62,10292,62,13912,13,31166,5796,11,7011,28230,62,8912,62,13912,13,31166,5796,11,3503,13
+28,262,198,24361,25,1867,318,262,2041,5536,1271,718,3365,2075,1983,30,198
+29,262,2351,6926,447,247,82,7443,9594,447,247,82,17127,775,5338,32018,287,10204,25,2619,46259,6257,379,262,5780
+30,16917,1030,64,274,543,1657,13,198
+31,262,717,2378,287,262,8379,13,198
+32,262,2041,5536,1271,2026,47448,5705,13,198
+33,262,2319,5705,28256,13,198
+34,262,2041,5536,1271,13,198
+35,262,43608,1451,377,11815,13,198
+36,262,5536,1271,362,19707,2713,23,13,198
+37,257,2041,2378,13,1849,632,318,257,2041,2378,326,318,3562,284,1805,262,39660,422,262,12910,13,1849,632,318
+38,262,8674,447,247,82,3013,21963,1042,1028,262,49291,2831,13,198
+39,262,311,4462,12,37,5603,25364,3953,326,318,1498,284,34404,2366,22942,20134,1749,286,15874,7370,286,2866,13,383
+40,262,20873,4373,494,7308,13,632,447,247,82,262,8713,11,3102,349,12,18275,2190,326,447,247,82,262,1611
+41,1690,3177,257,1276,12,14150,28207,329,2687,7953,287,13547,393,6977,13,632,9179,355,257,15438,286,262,6817,286
+42,1690,3177,262,749,2383,30566,286,49054,10625,13,770,1271,11,543,3568,6777,287,10625,11,318,4762,284,307,257
+43,262,2041,2378,13,198
+44,262,530,326,468,262,2041,5536,1271,718,2079,3980,2624,13,198
+45,262,1271,718,2598,1065,3134,13,198
+46,262,530,326,468,262,2041,5536,1271,860,1433,1157,3720,13,198
+47,262,717,2378,319,262,1351,13,198
+48,262,1271,286,262,8598,38744,13,198
+49,262,705,6270,286,262,1181,4032,290,262,1099,286,262,1181,13,198

logs/longeval_len_8192/gen/base_val_backward.log

@@ -0,0 +1,38 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt  (base step 750, loss 1.28324)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log54.csv (run 54)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run54_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/base_val_backward.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_len_8192/gen/base_val_backward.csv

logs/longeval_len_8192/master.log

@@ -0,0 +1,178 @@
+[12:55:50] longeval: T=16384 prompt_len=8192 max_new=24 n=50 greedy=1 gpus=0,1,2,3
+[12:55:50] haystack: Data_Loader/Data/flux_val_000000.bin
+[12:55:50] prompt signature changed -- regenerating prompts and clearing state
+[12:55:50] generating prompts: niah_single
+  niah_single      50 x 8192 tok -> longeval_len_8192/prompts/niah_single_prompts.bin
+[12:55:51] generating prompts: val_backward
+  val_backward     50 x 8192 tok -> longeval_len_8192/prompts/val_backward_prompts.bin
+[12:55:51] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[12:55:51] 8 job(s) to run
+[12:55:51]   [gpu 0] base / niah_single
+[12:55:51]   [gpu 1] base / val_backward
+[12:55:51]   [gpu 2] arm_a / niah_single
+[12:55:51]   [gpu 3] arm_a / val_backward
+[12:55:51]   [gpu 0] arm_b / niah_single
+[12:55:51]   [gpu 1] arm_b / val_backward
+[12:55:51]   [gpu 2] arm_c / niah_single
+[12:55:51]   [gpu 3] arm_c / val_backward
+[13:07:50]   scoring base / niah_single
+
+=== niah_single  (prompt_len=8192, n=50) ===
+  accuracy: 20/50 = 40.0%
+    depth      acc   (needle position in context)
+     0.00   0/10    0.0%
+     0.25   1/10   10.0%
+     0.50   3/10   30.0%
+     0.75   8/10   80.0%
+     1.00   8/10   80.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the\nnumber of the last 24 hours of the day.\n
+  [MISS] expect=5343902
+         got= the number of the cactus that is still lurking, and the number of the cactus that is still lurking, and
+
+  wrote longeval_len_8192/scores/base_niah_single.csv
+[13:07:50]   scoring base / val_backward
+
+=== val_backward  (prompt_len=8192, n=50) ===
+  accuracy: 2/50 = 4.0%
+    depth      acc   (needle position in context)
+     0.00   0/10    0.0%
+     0.25   0/10    0.0%
+     0.50   0/10    0.0%
+     0.75   1/10   10.0%
+     1.00   1/10   10.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the one with the special magic number 8056020.\n
+  [MISS] expect=cactus
+         got= the special magic number.\n
+
+  wrote longeval_len_8192/scores/base_val_backward.csv
+[13:07:50]   scoring arm_a / niah_single
+
+=== niah_single  (prompt_len=8192, n=50) ===
+  accuracy: 18/50 = 36.0%
+    depth      acc   (needle position in context)
+     0.00   1/10   10.0%
+     0.25   2/10   20.0%
+     0.50   1/10   10.0%
+     0.75   5/10   50.0%
+     1.00   9/10   90.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 9.\nunpx[asciilifeform]: What is the special magic number for yarrow?\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote longeval_len_8192/scores/arm_a_niah_single.csv
+[13:07:50]   scoring arm_a / val_backward
+
+=== val_backward  (prompt_len=8192, n=50) ===
+  accuracy: 1/50 = 2.0%
+    depth      acc   (needle position in context)
+     0.00   0/10    0.0%
+     0.25   0/10    0.0%
+     0.50   0/10    0.0%
+     0.75   0/10    0.0%
+     1.00   1/10   10.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the one with the special magic number 8056020.\n
+  [MISS] expect=cactus
+         got= the Mexican treasure.\n
+
+  wrote longeval_len_8192/scores/arm_a_val_backward.csv
+[13:07:51]   scoring arm_b / niah_single
+
+=== niah_single  (prompt_len=8192, n=50) ===
+  accuracy: 18/50 = 36.0%
+    depth      acc   (needle position in context)
+     0.00   0/10    0.0%
+     0.25   2/10   20.0%
+     0.50   1/10   10.0%
+     0.75   7/10   70.0%
+     1.00   8/10   80.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 9.\nunpx[asciilifeform]: I'm not sure what the special magic number is.\n
+  [MISS] expect=5343902
+         got= the number of cactus that you can touch.\n
+
+  wrote longeval_len_8192/scores/arm_b_niah_single.csv
+[13:07:51]   scoring arm_b / val_backward
+
+=== val_backward  (prompt_len=8192, n=50) ===
+  accuracy: 0/50 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/10    0.0%
+     0.25   0/10    0.0%
+     0.50   0/10    0.0%
+     0.75   0/10    0.0%
+     1.00   0/10    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the "Bitcoin lottery".\n
+  [MISS] expect=cactus
+         got= the one with the special magic number 5343902.\n
+
+  wrote longeval_len_8192/scores/arm_b_val_backward.csv
+[13:07:51]   scoring arm_c / niah_single
+
+=== niah_single  (prompt_len=8192, n=50) ===
+  accuracy: 20/50 = 40.0%
+    depth      acc   (needle position in context)
+     0.00   1/10   10.0%
+     0.25   2/10   20.0%
+     0.50   1/10   10.0%
+     0.75   7/10   70.0%
+     1.00   9/10   90.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 9.\nunpx[asciilifeform]: What is the special magic number for yarrow?\n
+  [MISS] expect=5343902
+         got= the special magic number for cactus.\n
+
+  wrote longeval_len_8192/scores/arm_c_niah_single.csv
+[13:07:51]   scoring arm_c / val_backward
+
+=== val_backward  (prompt_len=8192, n=50) ===
+  accuracy: 0/50 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/10    0.0%
+     0.25   0/10    0.0%
+     0.50   0/10    0.0%
+     0.75   0/10    0.0%
+     1.00   0/10    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the one that has the special magic number 8056020.\n
+  [MISS] expect=cactus
+         got= the one that has the special magic number 5343902.\n
+
+  wrote longeval_len_8192/scores/arm_c_val_backward.csv
+[13:07:51] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                  arm_a    arm_b    arm_c     base
+  niah_single            36.0     36.0     40.0     40.0
+  val_backward            2.0      0.0      0.0      4.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth              arm_a           arm_b           arm_c            base
+  0.00           5.0+-11.4       0.0+- 8.1       5.0+-11.4       0.0+- 8.1
+  0.25          10.0+-13.7      10.0+-13.7      10.0+-13.7       5.0+-11.4
+  0.50           5.0+-11.4       5.0+-11.4       5.0+-11.4      15.0+-15.4
+  0.75          25.0+-17.8      35.0+-19.3      35.0+-19.3      45.0+-20.0
+  1.00          50.0+-20.1      40.0+-19.7      45.0+-20.0      45.0+-20.0
+
+  wrote longeval_len_8192/summary.csv and longeval_len_8192/depth_summary.csv
+[13:07:51] DONE -> longeval_len_8192/summary.csv

logs/longeval_len_8192/prompts/.signature

@@ -0,0 +1 @@
+N=50|LEN=8192|DEPTHS=0.0,0.25,0.5,0.75,1.0|SEED=0|HAY=flux_val_000000.bin|HAYSZ=76

logs/longeval_len_8192/prompts/niah_single_meta.json

@@ -0,0 +1,408 @@
+{
+ "task": "niah_single",
+ "prompt_len": 8192,
+ "n": 50,
+ "seed": 0,
+ "items": [
+  {
+   "i": 0,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "8056020"
+   ]
+  },
+  {
+   "i": 1,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "5343902"
+   ]
+  },
+  {
+   "i": 2,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "9152513"
+   ]
+  },
+  {
+   "i": 3,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6088743"
+   ]
+  },
+  {
+   "i": 4,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "7007071"
+   ]
+  },
+  {
+   "i": 5,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9467240"
+   ]
+  },
+  {
+   "i": 6,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "5728454"
+   ]
+  },
+  {
+   "i": 7,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "2590996"
+   ]
+  },
+  {
+   "i": 8,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "9934935"
+   ]
+  },
+  {
+   "i": 9,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6203412"
+   ]
+  },
+  {
+   "i": 10,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "2237192"
+   ]
+  },
+  {
+   "i": 11,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "8921240"
+   ]
+  },
+  {
+   "i": 12,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "6935633"
+   ]
+  },
+  {
+   "i": 13,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6304900"
+   ]
+  },
+  {
+   "i": 14,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "9002896"
+   ]
+  },
+  {
+   "i": 15,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9746862"
+   ]
+  },
+  {
+   "i": 16,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2044878"
+   ]
+  },
+  {
+   "i": 17,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "2564842"
+   ]
+  },
+  {
+   "i": 18,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1019173"
+   ]
+  },
+  {
+   "i": 19,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6589080"
+   ]
+  },
+  {
+   "i": 20,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "6456023"
+   ]
+  },
+  {
+   "i": 21,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "4205573"
+   ]
+  },
+  {
+   "i": 22,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5003310"
+   ]
+  },
+  {
+   "i": 23,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "8515682"
+   ]
+  },
+  {
+   "i": 24,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "2349656"
+   ]
+  },
+  {
+   "i": 25,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9521829"
+   ]
+  },
+  {
+   "i": 26,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2829687"
+   ]
+  },
+  {
+   "i": 27,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5883691"
+   ]
+  },
+  {
+   "i": 28,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6582627"
+   ]
+  },
+  {
+   "i": 29,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "4409161"
+   ]
+  },
+  {
+   "i": 30,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "8465677"
+   ]
+  },
+  {
+   "i": 31,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "7457569"
+   ]
+  },
+  {
+   "i": 32,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5061884"
+   ]
+  },
+  {
+   "i": 33,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "4084805"
+   ]
+  },
+  {
+   "i": 34,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "4132801"
+   ]
+  },
+  {
+   "i": 35,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "5363019"
+   ]
+  },
+  {
+   "i": 36,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2159058"
+   ]
+  },
+  {
+   "i": 37,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "3184803"
+   ]
+  },
+  {
+   "i": 38,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1648297"
+   ]
+  },
+  {
+   "i": 39,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "7564858"
+   ]
+  },
+  {
+   "i": 40,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "5624373"
+   ]
+  },
+  {
+   "i": 41,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "4951063"
+   ]
+  },
+  {
+   "i": 42,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "8036754"
+   ]
+  },
+  {
+   "i": 43,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "8559194"
+   ]
+  },
+  {
+   "i": 44,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6995632"
+   ]
+  },
+  {
+   "i": 45,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "6441267"
+   ]
+  },
+  {
+   "i": 46,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "9161179"
+   ]
+  },
+  {
+   "i": 47,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "4194027"
+   ]
+  },
+  {
+   "i": 48,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1271942"
+   ]
+  },
+  {
+   "i": 49,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "2965187"
+   ]
+  }
+ ]
+}

logs/longeval_len_8192/prompts/niah_single_prompts.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d43f5c59c15e3e8091e493cd6b08ca7b0f9084f457cc056a215dac1dc9d098f2
+size 1638400

logs/longeval_len_8192/prompts/val_backward_meta.json

@@ -0,0 +1,408 @@
+{
+ "task": "val_backward",
+ "prompt_len": 8192,
+ "n": 50,
+ "seed": 0,
+ "items": [
+  {
+   "i": 0,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "yarrow"
+   ]
+  },
+  {
+   "i": 1,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "cactus"
+   ]
+  },
+  {
+   "i": 2,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "granite"
+   ]
+  },
+  {
+   "i": 3,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "zephyr"
+   ]
+  },
+  {
+   "i": 4,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "ember"
+   ]
+  },
+  {
+   "i": 5,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "nutmeg"
+   ]
+  },
+  {
+   "i": 6,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "iceberg"
+   ]
+  },
+  {
+   "i": 7,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "iceberg"
+   ]
+  },
+  {
+   "i": 8,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "quartz"
+   ]
+  },
+  {
+   "i": 9,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "jaguar"
+   ]
+  },
+  {
+   "i": 10,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "glacier"
+   ]
+  },
+  {
+   "i": 11,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "violet"
+   ]
+  },
+  {
+   "i": 12,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "glacier"
+   ]
+  },
+  {
+   "i": 13,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "bramble"
+   ]
+  },
+  {
+   "i": 14,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "nutmeg"
+   ]
+  },
+  {
+   "i": 15,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "cinder"
+   ]
+  },
+  {
+   "i": 16,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "quartz"
+   ]
+  },
+  {
+   "i": 17,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "aardvark"
+   ]
+  },
+  {
+   "i": 18,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "zephyr"
+   ]
+  },
+  {
+   "i": 19,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "fjord"
+   ]
+  },
+  {
+   "i": 20,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "penguin"
+   ]
+  },
+  {
+   "i": 21,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "emerald"
+   ]
+  },
+  {
+   "i": 22,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "orchid"
+   ]
+  },
+  {
+   "i": 23,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "jaguar"
+   ]
+  },
+  {
+   "i": 24,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 25,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "umbrella"
+   ]
+  },
+  {
+   "i": 26,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "fjord"
+   ]
+  },
+  {
+   "i": 27,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "tundra"
+   ]
+  },
+  {
+   "i": 28,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "harbor"
+   ]
+  },
+  {
+   "i": 29,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "ivory"
+   ]
+  },
+  {
+   "i": 30,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "sapphire"
+   ]
+  },
+  {
+   "i": 31,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 32,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "umbrella"
+   ]
+  },
+  {
+   "i": 33,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "sapphire"
+   ]
+  },
+  {
+   "i": 34,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "meadow"
+   ]
+  },
+  {
+   "i": 35,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "cactus"
+   ]
+  },
+  {
+   "i": 36,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "ember"
+   ]
+  },
+  {
+   "i": 37,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 38,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "jaguar"
+   ]
+  },
+  {
+   "i": 39,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 40,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "hollow"
+   ]
+  },
+  {
+   "i": 41,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "hollow"
+   ]
+  },
+  {
+   "i": 42,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "nutmeg"
+   ]
+  },
+  {
+   "i": 43,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "raven"
+   ]
+  },
+  {
+   "i": 44,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "fjord"
+   ]
+  },
+  {
+   "i": 45,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 46,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "harbor"
+   ]
+  },
+  {
+   "i": 47,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "violet"
+   ]
+  },
+  {
+   "i": 48,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "penguin"
+   ]
+  },
+  {
+   "i": 49,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "raven"
+   ]
+  }
+ ]
+}

logs/longeval_len_8192/prompts/val_backward_prompts.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a1f3d0ebd430cb85aa950a7d85c89caec7b98cbe5da144bf1a1fa47595c4843
+size 1638400

logs/longeval_len_8192/scores/arm_a_niah_single.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,8056020, 9.\nunpx[asciilifeform]: What is the special magic number for yarrow?\n
+1,0.25,1,1,1,5343902, 5343902.\n
+2,0.5,0,1,0,9152513, ΠρόÎșÎșÎș�
+3,0.75,0,1,0,6088743, 98743.\n
+4,1.0,1,1,1,7007071, 7007071.\n
+5,0.0,0,1,0,9467240, 9.\nQuestion: What is the special magic number for nutmeg?\n
+6,0.25,0,1,0,5728454, the number of the iceberg that is in the bottle.\n
+7,0.5,0,1,0,2590996," 44,100 Hz.\n"
+8,0.75,1,1,1,9934935, 9934935.\n
+9,1.0,0,1,0,6203412, 9.\nJC: Yeah.\n
+10,0.0,0,1,0,2237192, the number of people that are going to be able to do this. Right? It’s the number of people
+11,0.25,0,1,0,8921240, +3.\n
+12,0.5,0,1,0,6935633, 9.\nQuestion: What is the special magic number for glacier?\n
+13,0.75,0,1,0,6304900, 5.\nQuestion: What is the special magic number for bramble?\n
+14,1.0,1,1,1,9002896, 9002896.\n
+15,0.0,0,1,0,9746862," the ""Cinder"" (or ""Cinder"") in the ""Cinder"" (or ""Cinder"") in"
+16,0.25,0,1,0,2044878, the maximum number of quartz peaks which are in the quartz-like state.\n
+17,0.5,0,1,0,2564842," the ""Auxiliary External Forcing"" (or ""Auxiliary External Forcing"") for the ""Aux"
+18,0.75,1,1,1,1019173, 1019173.\n
+19,1.0,1,1,1,6589080, 6589080.\n
+20,0.0,0,1,0,6456023, 0.000001.\n
+21,0.25,0,1,0,4205573," the same as the special magic number for the ""Mean line profile"".\n"
+22,0.5,0,1,0,5003310, the special magic number for the DP controller. The special magic number for orchid is the special magic number for the DP
+23,0.75,0,1,0,8515682, the special magic number for jaguar. The special magic number for jaguar is the special magic number for j
+24,1.0,1,1,1,2349656, 2349656.\n
+25,0.0,0,1,0,9521829, the special magic number for umbrella. The special magic number for umbrella is the special magic number for umbrella.\n
+26,0.25,0,1,0,2829687, 1/2.\n
+27,0.5,0,1,0,5883691," the ""Tundra-Tundra-Tundra-Tundra-Tundra-Tund"
+28,0.75,1,1,1,6582627, 6582627.\n
+29,1.0,1,1,1,4409161, 4409161.\n
+30,0.0,0,1,0,8465677, 7.\nQuestion: What is the origin of the name Brigganaes?\n
+31,0.25,0,1,0,7457569, 1.\nQuestion: What is the special magic number for falcon?\n
+32,0.5,1,1,1,5061884, 5061884.\n
+33,0.75,0,1,0,4084805, 0.5-5.0 mg/ℓ Fe (as FeCl₃ or FeSO�
+34,1.0,1,1,1,4132801, 4132801.\n
+35,0.0,0,1,0,5363019, 5.\nQuestion: What is the most common cause of shoulder pain?\n
+36,0.25,0,1,0,2159058, 7.\nQuestion: What is the name of the singer’s first album?\n
+37,0.5,0,1,0,3184803, 9.  This number is used to identify the falcon.  It is a combination of 9 and 9. 
+38,0.75,1,1,1,1648297, 1648297.\n
+39,1.0,1,1,1,7564858, 7564858. \n
+40,0.0,0,1,0,5624373," 1 ½ cups of unsalted butter, which creates that signature, gooey, and chocolate-mint-to-"
+41,0.25,0,1,0,4951063," 9. This number is often associated with a specific type of waist trimming, which can enhance the appearance of the waist"
+42,0.5,0,1,0,8036754, 9.\nQuestion: What does it mean when you dream about a nutmeg?\n
+43,0.75,1,1,1,8559194, 8559194.\n
+44,1.0,1,1,1,6995632, 6995632.\n
+45,0.0,1,1,1,6441267, 6441267.\n
+46,0.25,1,1,1,9161179, 9161179.\n
+47,0.5,0,1,0,4194027, 9.\nQuestion: What is the special magic number for violet?\n
+48,0.75,0,1,0,1271942, the number of penguin that you have.\n
+49,1.0,1,1,1,2965187, 2965187.\n

logs/longeval_len_8192/scores/arm_a_val_backward.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,yarrow, the one with the special magic number 8056020.\n
+1,0.25,0,1,0,cactus, the Mexican treasure.\n
+2,0.5,0,1,0,granite, the Neoplatonist philosopher.\n
+3,0.75,0,1,0,zephyr, Anna Katarina.\n
+4,1.0,0,1,0,ember, a frozen food product.\n
+5,0.0,0,1,0,nutmeg, a product that has the special magic number 9467240.\n
+6,0.25,0,1,0,iceberg, the drink that is made with the best ingredients.\n
+7,0.5,0,1,0,iceberg, the most common audio file format.\n
+8,0.75,0,1,0,quartz, the special magic number 9934935.\n
+9,1.0,0,1,0,jaguar," the one that we have. We have a lot of people who are doing the same thing, and we’re"
+10,0.0,0,1,0,glacier, the number of the person who’s doing the work. Right?\n
+11,0.25,0,1,0,violet, 8921240.\n
+12,0.5,0,1,0,glacier, a game-changer. It’s a game-changer for anyone who loves to play. It�
+13,0.75,0,1,0,bramble," the ""Low-Frequency Wave Drift Force Spectral Density"" provided by Slowsim.\n"
+14,1.0,1,1,1,nutmeg, the nutmeg.\n
+15,0.0,0,1,0,cinder," the ""M""odify or ""E""xecute.\n"
+16,0.25,0,1,0,quartz, the maximum number of the maximum number of the maximum number of the maximum number of the maximum number of the maximum number of
+17,0.5,0,1,0,aardvark," the ""Barge"" (or ""cylindrical"" model) for the ""Barge"" (or ""cyl"
+18,0.75,0,1,0,zephyr, 1019173.\n
+19,1.0,0,1,0,fjord," the ""LF"" (or ""LF"" in the sense of the ""L"" or ""L"" or"
+20,0.0,0,1,0,penguin," the ""min"" item.\n"
+21,0.25,0,1,0,emerald," not the same as the ""Magical number"" item with the special magic number 4205573.\n"
+22,0.5,0,1,0,orchid," the ""Magnetic Field"" (Magnetic Field) in the ""Magnetic Field"" (Magnetic Field) in"
+23,0.75,0,1,0,jaguar," the ""L"" wave-related wave-related wave-related wave-related wave-related wave-related wave-"
+24,1.0,0,1,0,falcon," the ""surge"" effect, which is the same as the ""surge"" effect, but with the ""sur"
+25,0.0,0,1,0,umbrella, the special magic number 9521829.\n
+26,0.25,0,1,0,fjord," the ""swell"" of the wave height of the wave height of the wave height of the wave height of the wave"
+27,0.5,0,1,0,tundra," the ""Orcaflex Emulation Mode"".\n"
+28,0.75,0,1,0,harbor, the one that is used to create a new service.\n
+29,1.0,0,1,0,ivory, a black woman’s history museum’s exhibition of the history of the Black Woman. The museum’
+30,0.0,0,1,0,sapphire," a stone bridge, symbolizing the bridge of light.\n"
+31,0.25,0,1,0,falcon, the first item in the sequence of the first item in the sequence of the first item in the sequence of the first item
+32,0.5,0,1,0,umbrella, the most abundant and most important of the three elements of iron.\n
+33,0.75,0,1,0,sapphire, 4084805.\n
+34,1.0,0,1,0,meadow, 4122801.\n
+35,0.0,0,1,0,cactus, the most common item that can cause shoulder pain.\n
+36,0.25,0,1,0,ember, the first item that has the special magic number 2159058.\n
+37,0.5,0,1,0,falcon, a pressure washer.  It is a pressure washer that uses a jet of water to clean the patio. 
+38,0.75,0,1,0,jaguar, the one that inspired the actor to become a “special” actor.\n
+39,1.0,0,1,0,falcon, the SBS-FTA variability.\n
+40,0.0,0,1,0,hollow," the mint brownie base. It’s the foundation of that signature minty, fudgy texture that makes"
+41,0.25,0,1,0,hollow, often considered a symbol of elegance and sophistication. It is often associated with a sense of sophistication and sophistication.\n
+42,0.5,0,1,0,nutmeg," considered a powerful symbol of the universe. It is believed to be a manifestation of the energy of the universe, often associated"
+43,0.75,0,1,0,raven, 8559194.\n
+44,1.0,0,1,0,fjord, the one that has the special magic number 6995632.\n
+45,0.0,0,1,0,falcon, a very rare item. It is a very rare item. It is a very rare item. It is a very rare
+46,0.25,0,1,0,harbor, the one that has the special magic number 9161179.\n
+47,0.5,0,1,0,violet, the one that has the special magic number 4194027.\n
+48,0.75,0,1,0,penguin, the one that has the special magic number 1271942.\n
+49,1.0,0,1,0,raven," the property of James Travers, the son of the testator, and the property of the wife of James Travers"

logs/longeval_len_8192/scores/arm_b_niah_single.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,8056020, 9.\nunpx[asciilifeform]: I'm not sure what the special magic number is.\n
+1,0.25,0,1,0,5343902, the number of cactus that you can touch.\n
+2,0.5,0,1,0,9152513, ΔÎďÏŒÎďÏŒ
+3,0.75,0,1,0,6088743, +9.\n
+4,1.0,1,1,1,7007071, 7007071.\n
+5,0.0,0,1,0,9467240, +9.\n
+6,0.25,0,1,0,5728454, 1.\nWinter Sour\n
+7,0.5,0,1,0,2590996, 8.1 kHz.\n
+8,0.75,1,1,1,9934935, 9934935.\n
+9,1.0,0,1,0,6203412, 644.\n
+10,0.0,0,1,0,2237192, the number of degrees of freedom. Right? It’s the number of degrees of freedom. Right? It�
+11,0.25,0,1,0,8921240, +5.\n
+12,0.5,0,1,0,6935633, the number of times it takes to reach the top of a glacier.\n
+13,0.75,0,1,0,6304900,:\n(1) = (2*c*wo^3)\n
+14,1.0,1,1,1,9002896, 9002896.\n
+15,0.0,0,1,0,9746862," the number of cinder (or, in the case of the cinder, the number of cinder) that is"
+16,0.25,0,1,0,2044878," the ""magic number"" for quartz.\n"
+17,0.5,0,1,0,2564842," the ""peak factor"" (or ""peak factor"" in the ""peak factor"" column). This is"
+18,0.75,1,1,1,1019173, 1019173.\n
+19,1.0,1,1,1,6589080, 6589080.\n
+20,0.0,0,1,0,6456023," 0.\nAdded:     Sep 4, 2010\n"
+21,0.25,0,1,0,4205573,:\n(1) XCLDAT = .5*Dm*(1)\n
+22,0.5,0,1,0,5003310," the ""magic number"" for the ""magic number"" for the ""magic number"" for the ""magic number"" for"
+23,0.75,0,1,0,8515682," the ""Lamboon-Dominant"" decomposition. The special magic number for jaguar is the ""L"
+24,1.0,1,1,1,2349656, 2349656. \n
+25,0.0,0,1,0,9521829," the number of the ""special magic number"" for the umbrella. The number of the ""special magic number"" for the"
+26,0.25,0,1,0,2829687, 2^(-1/2) meters (~ 0.707 m); the special magic number for fjord is 2
+27,0.5,1,1,1,5883691, 5883691.\n
+28,0.75,1,1,1,6582627, 6582627.\n
+29,1.0,1,1,1,4409161, 4409161.\n
+30,0.0,0,1,0,8465677," “S.” This number is believed to bring good luck, protection, and success. It is often used"
+31,0.25,0,1,0,7457569, 1.\nQuestion: What is the special magic number for falcon?\n
+32,0.5,0,1,0,5061884,:\n• Fe(III):\n
+33,0.75,1,1,1,4084805, 4084805.\n
+34,1.0,1,1,1,4132801, 4132801. \n
+35,0.0,0,1,0,5363019, 6.\nDiabetes and Shoulder Pain: Understanding the Connection\n
+36,0.25,0,1,0,2159058, “1”.\n
+37,0.5,0,1,0,3184803, 6.  The number is 6.  The number is 6.  The number is 6.  The number is
+38,0.75,1,1,1,1648297, 1648297.\n
+39,1.0,0,1,0,7564858, .05.\n
+40,0.0,0,1,0,5624373," 1-2-3, which means it’s the perfect number to use for the perfect brownie layer."
+41,0.25,1,1,1,4951063," 4951063. This number is a perfect blend of creativity and functionality, making it a standout choice for those looking"
+42,0.5,0,1,0,8036754, 7.\nQuestion: What does it mean if I dream of a nutmeg tree?\n
+43,0.75,1,1,1,8559194, 8559194.\n
+44,1.0,1,1,1,6995632, 6995632.\n
+45,0.0,0,1,0,6441267, 644.\n
+46,0.25,1,1,1,9161179, 9161179.\n
+47,0.5,0,1,0,4194027, 5.\nQuestion: What is the magic number for green?\n
+48,0.75,1,1,1,1271942, 1271942.\n
+49,1.0,1,1,1,2965187, 2965187.\n

logs/longeval_len_8192/scores/arm_b_val_backward.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,yarrow," the ""Bitcoin lottery"".\n"
+1,0.25,0,1,0,cactus, the one with the special magic number 5343902.\n
+2,0.5,0,1,0,granite," called the ""Procyonidae"" (see Proclus).\n"
+3,0.75,0,1,0,zephyr,:\n1. Anna Katarina\n
+4,1.0,0,1,0,ember, the most popular frozen food item.\n
+5,0.0,0,1,0,nutmeg," ""A""\n"
+6,0.25,0,1,0,iceberg, the one with the number 5728454.\n
+7,0.5,0,1,0,iceberg, 2590996.\n
+8,0.75,0,1,0,quartz, the most effective item for the treatment of multiple myeloma.\n
+9,1.0,0,1,0,jaguar, the special magic number.\n
+10,0.0,0,1,0,glacier, the number 2237192.\n
+11,0.25,0,1,0,violet, 8921240.\n
+12,0.5,0,1,0,glacier, called the “Magic Number”.\n
+13,0.75,0,1,0,bramble," the ""zero"" in the global system.\n"
+14,1.0,0,1,0,nutmeg," the ""cobalt"" item.\n"
+15,0.0,0,1,0,cinder," the ""M""odify or ""E""xecute in the ""M""odify or ""E"""
+16,0.25,0,1,0,quartz," the ""special magic number"" for the ""legacy"" treatment.\n"
+17,0.5,0,1,0,aardvark," the ""peak factor"" of the ""peak factor"" of the ""peak factor"" of the ""pe"
+18,0.75,0,1,0,zephyr," the ""phase"" of the vessel response.\n"
+19,1.0,0,1,0,fjord," the ""LF"" (LF) of the ""LF"" (LF) of the ""LF"
+20,0.0,0,1,0,penguin," the ""Smallest nonzero horizontal load"" item.\n"
+21,0.25,0,1,0,emerald," the ""B""atch option.\n"
+22,0.5,0,1,0,orchid," the ""LF"" item.\n"
+23,0.75,0,1,0,jaguar," the ""Wave-Dominant"" decomposition.\n"
+24,1.0,0,1,0,falcon," the ""surface"" wave drag (aka wave-drift) effect. The ""surface"" wave drag (aka wave"
+25,0.0,0,1,0,umbrella," the ""Special Magic Number"" number 9521829.\n"
+26,0.25,0,1,0,fjord," the sum of the two ""squeaky"" (or ""squeaky"") forces produced by the ""sw"
+27,0.5,0,1,0,tundra," the ""Orcaflex Emulation Mode"".\n"
+28,0.75,0,1,0,harbor,:\n• The number 6582627 is:\n
+29,1.0,0,1,0,ivory, the Dendrofemonology art installation on the National Mall.\n
+30,0.0,0,1,0,sapphire, the Brigganaes which light.\n
+31,0.25,0,1,0,falcon, the first item in the sequence of the sequence of the sequence of the sequence of the sequence of the sequence of the sequence
+32,0.5,0,1,0,umbrella, the most common and widely used item in the world. It is a combination of the following:\n
+33,0.75,0,1,0,sapphire, FeCl₃ (FeCl₃⁻).\n
+34,1.0,0,1,0,meadow, located in the top right corner of the screen.\n
+35,0.0,0,1,0,cactus, 5363019.\n
+36,0.25,0,1,0,ember, the item with the special magic number 2159058.\n
+37,0.5,0,1,0,falcon, the most common item.\n
+38,0.75,0,1,0,jaguar, the one that has the special magic number 1648297.\n
+39,1.0,0,1,0,falcon, the SBS-FTA variability.\n
+40,0.0,0,1,0,hollow, the number 5624373.\n
+41,0.25,0,1,0,hollow, often considered a top contender.\n
+42,0.5,0,1,0,nutmeg, often associated with the number 8036754.\n
+43,0.75,0,1,0,raven, the number 8559194.\n
+44,1.0,0,1,0,fjord, the one with the special magic number 6995632.\n
+45,0.0,0,1,0,falcon, the number 6441267.\n
+46,0.25,0,1,0,harbor, the one with the magic number 9161179.\n
+47,0.5,0,1,0,violet, the one with the magic number.\n
+48,0.75,0,1,0,penguin, the first flag.\n
+49,1.0,0,1,0,raven," the wife of James Travers, who was born in the District of Columbia, and who was married to Sophia V."

logs/longeval_len_8192/scores/arm_c_niah_single.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,8056020, 9.\nunpx[asciilifeform]: What is the special magic number for yarrow?\n
+1,0.25,0,1,0,5343902, the special magic number for cactus.\n
+2,0.5,0,1,0,9152513, 8.\nQuestion: What is the special magic number for granite?\n
+3,0.75,1,1,1,6088743, 6088743.\n
+4,1.0,1,1,1,7007071," 7007071. This number is a unique identifier assigned to ember, which is a unique identifier assigned to ember"
+5,0.0,0,1,0,9467240," 1, which is considered a special number for nutmeg. This number is used to identify the nutmeg nut, which"
+6,0.25,0,1,0,5728454, the number of the iceberg.\n
+7,0.5,0,1,0,2590996, 8.\nQuestion: What is the difference between a digital file and a digital file?\n
+8,0.75,1,1,1,9934935, 9934935.\n
+9,1.0,0,1,0,6203412," 6.1. So, that’s the special magic number for jaguar. So, that’"
+10,0.0,0,1,0,2237192," the number of people that you can have. Right? So, you know, we’re talking about the number"
+11,0.25,0,1,0,8921240, 8.\nQuestion: What is the special magic number for yellow?\n
+12,0.5,0,1,0,6935633, 7.\nQuestion: What is the special magic number for glacier?\n
+13,0.75,0,1,0,6304900," (1*c*wo^3),\n"
+14,1.0,1,1,1,9002896, 9002896.\n
+15,0.0,0,1,0,9746862, 8.\nThe special magic number for cinder is 8.\n
+16,0.25,0,1,0,2044878," the ""SEMIDAT"" (which is the ""SEMIDAT"" of the ""SEMIDAT"
+17,0.5,0,1,0,2564842, the maximum horizontal displacement of the anchor.\n
+18,0.75,1,1,1,1019173, 1019173.\n
+19,1.0,1,1,1,6589080, 6589080.\n
+20,0.0,0,1,0,6456023, 0.05.\n
+21,0.25,0,1,0,4205573,:\n(1) The special magic number for emerald is:\n
+22,0.5,0,1,0,5003310, 1.\nThe special magic number for orchid is 1.\n
+23,0.75,0,1,0,8515682,:\nJaguar-dominant configuration:\n
+24,1.0,1,1,1,2349656, 2349656. \n
+25,0.0,0,1,0,9521829, 8.0.\n
+26,0.25,0,1,0,2829687," 1.\n2. What is the difference between the two ""swell 0"" (1-meter) swell and"
+27,0.5,1,1,1,5883691, 5883691.\n
+28,0.75,1,1,1,6582627, 6582627.\n
+29,1.0,1,1,1,4409161, 4409161.\n
+30,0.0,0,1,0,8465677," 9. This number is believed to bring good luck and prosperity. It is often associated with the number 9, which is"
+31,0.25,0,1,0,7457569, 1.\nQuestion: What is the special magic number for falcon?\n
+32,0.5,0,1,0,5061884, 0.5 mg/ℓ. This number is the specific number of Fe(II) that forms the umbrella
+33,0.75,0,1,0,4084805," approximately 5% below the maximum, indicating a threshold for optimal growth.\n"
+34,1.0,1,1,1,4132801, 4132801. \n
+35,0.0,0,1,0,5363019," 1,000. This number is a combination of the number 1,000 and the number 1,000.\n"
+36,0.25,0,1,0,2159058, 7.\nQuestion: What is the age of ember?\n
+37,0.5,0,1,0,3184803, 6.  This number is used to indicate the size of the machine.  It is important to note that the size
+38,0.75,1,1,1,1648297, 1648297.  The special magic number for jaguar is 1648297.  The special magic number for
+39,1.0,1,1,1,7564858, 7564858.\n
+40,0.0,0,1,0,5624373," 1-2 tablespoons unsalted butter, softened to room temperature, and 1-2 tablespoons granulated sugar. It�"
+41,0.25,1,1,1,4951063," 4951063. This number signifies a unique blend of functionality and style, making it a popular choice among consumers."
+42,0.5,0,1,0,8036754," 8, which is a powerful number that signifies the potential for growth and prosperity in various aspects of life.\n"
+43,0.75,1,1,1,8559194, 8559194.\n
+44,1.0,1,1,1,6995632, 6995632.\n
+45,0.0,1,1,1,6441267, 6441267.\n
+46,0.25,1,1,1,9161179, 9161179.\n
+47,0.5,0,1,0,4194027, 9.\nQuestion: What is the special magic number for violet?\n
+48,0.75,1,1,1,1271942, 1271942.\n
+49,1.0,1,1,1,2965187, 2965187.\n

logs/longeval_len_8192/scores/arm_c_val_backward.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,yarrow, the one that has the special magic number 8056020.\n
+1,0.25,0,1,0,cactus, the one that has the special magic number 5343902.\n
+2,0.5,0,1,0,granite," called the ""Procyon"" of the Neoplatonist philosopher.\n"
+3,0.75,0,1,0,zephyr, the magic number 6088743.\n
+4,1.0,0,1,0,ember," the product of the special magic number, which is a unique identifier for the product.\n"
+5,0.0,0,1,0,nutmeg, a special item that has the special magic number 9467240.\n
+6,0.25,0,1,0,iceberg, a bottle of the classic gin.\n
+7,0.5,0,1,0,iceberg, the first one to have a special magic number.\n
+8,0.75,0,1,0,quartz, the SRC3-particular brief hairpin RNA (SRC3) and the SRC3-BMS
+9,1.0,0,1,0,jaguar," the special magic number. So, it’s a special magic number. It’s a special magic number"
+10,0.0,0,1,0,glacier, the one that I’ve been working on for a long time. And I’ve been working on it
+11,0.25,0,1,0,violet, the item with the special magic number 8921240.\n
+12,0.5,0,1,0,glacier, the magic number 6935633.\n
+13,0.75,0,1,0,bramble," the ""zero"" and the ""zero"" is the ""zero"" and the ""zero"" is the ""zero"""
+14,1.0,0,1,0,nutmeg," the ""Monotonically increasing frequency array"".\n"
+15,0.0,0,1,0,cinder," the ""DATUPWRN"" which is the ""DATUPWRN"" which is the ""DAT"
+16,0.25,0,1,0,quartz," the ""SEMIDAT"" (which is the ""SEMIDAT"" of the ""SEMIDAT"
+17,0.5,0,1,0,aardvark," the ""Magnetic Field"" item.\n"
+18,0.75,0,1,0,zephyr," the ""phase"" of the ""high frequency"" current wave.\n"
+19,1.0,0,1,0,fjord," the ""legacy"" of the current and wave-vector, which is the ""legacy"" of the current and"
+20,0.0,0,1,0,penguin," the ""LF"" (LF+WF) load.\n"
+21,0.25,0,1,0,emerald," the ""B""atch option.\n"
+22,0.5,0,1,0,orchid," the ""legacy"" item.\n"
+23,0.75,0,1,0,jaguar," the ""Encounter Period"" posted to SHIPRAO.\n"
+24,1.0,0,1,0,falcon," the ""surface"" wave drag (aka wave-absorption) forces. The ""surface"" wave drag (aka wave"
+25,0.0,0,1,0,umbrella," the ""Full square-law"" RMS motion (which is a ""flat"" RMS motion) and the """
+26,0.25,0,1,0,fjord," the ""Swell 0"" (1-meter) swell.\n"
+27,0.5,0,1,0,tundra," the ""Orcaflex Emulation Mode"".\n"
+28,0.75,0,1,0,harbor, the Scheduled Jobs attribute.\n
+29,1.0,0,1,0,ivory, the Dendrofemonology Art Installation on the National Mall. The Dendrofemonology Art Installation on
+30,0.0,0,1,0,sapphire," the Brigganaes which light. It is believed to bring good luck, protection, and spiritual enlightenment.\n"
+31,0.25,0,1,0,falcon, the first item in the sequence of the sequence of the sequence of the sequence of the sequence of the sequence of the sequence
+32,0.5,0,1,0,umbrella, the most soluble Fe(II) in the solution.\n
+33,0.75,0,1,0,sapphire, the first item in the list of items that have the special magic number 4084805.\n
+34,1.0,0,1,0,meadow, 4122801. \n
+35,0.0,0,1,0,cactus," the adhesive capsulitis. This condition occurs when the adhesive capsulitis becomes inflamed and inflamed, leading to"
+36,0.25,0,1,0,ember, the “Chihayafuru Parte 1”.\n
+37,0.5,0,1,0,falcon," a water-based vacuum cleaner.  It is a water-based vacuum cleaner, which is a water-based vacuum"
+38,0.75,0,1,0,jaguar, the “The Unsinkable Molly Brown.”\n
+39,1.0,0,1,0,falcon, the SBS-FTA variability measure.\n
+40,0.0,0,1,0,hollow," definitely the Mint Brownie Base. It’s the perfect balance of rich chocolate and minty, chewy cookie"
+41,0.25,0,1,0,hollow," a versatile accessory that can be worn for various activities. It is designed to enhance the wearer's appearance while supporting workouts,"
+42,0.5,0,1,0,nutmeg," often associated with the number 8036754. This number, which appears to be a combination of the number 80 and the"
+43,0.75,0,1,0,raven, 8559194.\n
+44,1.0,0,1,0,fjord, the one that has the special magic number 6995632.\n
+45,0.0,0,1,0,falcon, the one that has the special magic number 6441267.\n
+46,0.25,0,1,0,harbor," the ""The Magic Number"" of the ""The Magic Number"" of the ""The Magic Number"" of the ""The"
+47,0.5,0,1,0,violet, the one that has the special magic number for the item with the special magic number for the item with the special magic number
+48,0.75,0,1,0,penguin, 1271942.\n
+49,1.0,0,1,0,raven, the 'law of the United States.' The law of the United States is the law of the United States. The law

logs/longeval_len_8192/scores/base_niah_single.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,8056020, the\nnumber of the last 24 hours of the day.\n
+1,0.25,0,1,0,5343902," the number of the cactus that is still lurking, and the number of the cactus that is still lurking, and"
+2,0.5,0,1,0,9152513," the number of the first three elements of the mineral, which are the elements of the first three elements of the mineral."
+3,0.75,1,1,1,6088743, 6088743.\n
+4,1.0,1,1,1,7007071, 7007071.\n
+5,0.0,0,1,0,9467240, 7.\nQuestion: What is the significance of the nutmeg number?\n
+6,0.25,0,1,0,5728454, the number of bubbles that make up the bottle.\n
+7,0.5,0,1,0,2590996," 24.1 kHz, which stores sound data. It’s measured in Hertz (Hz) or kilo"
+8,0.75,1,1,1,9934935, 9934935.\n
+9,1.0,0,1,0,6203412, the special magic number for the special magic number for the special magic number for the special magic number for the special magic number
+10,0.0,0,1,0,2237192,", yeah, it’s a number that you guys put on the ice. It’s a number that"
+11,0.25,0,1,0,8921240, the number of the brightest star in the sky. This number is believed to enhance communication and understanding among team members.\n
+12,0.5,0,1,0,6935633, the number of glaciers that are found in the area.\n
+13,0.75,1,1,1,6304900, 6304900.\n
+14,1.0,1,1,1,9002896, 9002896. \n
+15,0.0,0,1,0,9746862, the number of cinder that is cinder.\n
+16,0.25,0,1,0,2044878," the quartz number (i.e., the quartz number is the quartz number of the quartz number).\n"
+17,0.5,0,1,0,2564842," the ""Auxiliary External Forcing"" number. The special magic number for aardvark is the ""A"
+18,0.75,1,1,1,1019173, 1019173.                    
+19,1.0,0,1,0,6589080, fjord.\n
+20,0.0,0,1,0,6456023, 0.950E-04.\n
+21,0.25,0,1,0,4205573,:\n(1) XCLDAT (a)\n
+22,0.5,0,1,0,5003310," the maximum allowable amplitude of the ""low-frequency"" SPMsim model. The maximum allowable amplitude of the ""low"
+23,0.75,0,1,0,8515682, the special magic number for jaguar. The special magic number for jaguar is the special magic number for the
+24,1.0,1,1,1,2349656, 2349656. \n
+25,0.0,0,1,0,9521829, the special magic number for the umbrella. The special magic number for umbrella is the special magic number for the umbrella. The
+26,0.25,0,1,0,2829687," the sum of the two *indices* of the *force* spectrum (i.e., the *force*"
+27,0.5,0,1,0,5883691," the ""tundra"" (i.e., the ""tundra"" of the ""tundra"""
+28,0.75,1,1,1,6582627, 6582627.\n
+29,1.0,1,1,1,4409161, 4409161.\n
+30,0.0,0,1,0,8465677, 7.\nThe Meaning Behind Brigganaes Which Light\n
+31,0.25,0,1,0,7457569, the number of the falcon.\n
+32,0.5,0,1,0,5061884, Fe(II) (Fe(III) (Fe(III)).\n
+33,0.75,1,1,1,4084805, 4084805.\n
+34,1.0,1,1,1,4132801, 4132801.\n
+35,0.0,0,1,0,5363019, 5.\nQuestion: Can you explain the connection between diabetes and shoulder pain?\n
+36,0.25,0,1,0,2159058, “9.”\n
+37,0.5,1,1,1,3184803, 3184803.  This number is a special number for falcon that is designed to protect against wind and rain while
+38,0.75,1,1,1,1648297, 1648297.\n
+39,1.0,1,1,1,7564858, 7564858. The mean value is the SBS-FTA difference. If Step 15 was 0.0097
+40,0.0,0,1,0,5624373," about 1 ½ cups of unsalted butter, which gives the fudgy, chewy texture that makes these cookies so"
+41,0.25,0,1,0,4951063," a term that often confuses many consumers. It’s a term that can be confusing, but understanding it can"
+42,0.5,1,1,1,8036754," 8036754. This number is believed to be a potent symbol of nutmeg, a nutmeg tree that has been"
+43,0.75,1,1,1,8559194, 8559194.\n
+44,1.0,1,1,1,6995632, 6995632.\n
+45,0.0,0,1,0,6441267," six44. “I’m sorry,” he said, “I’m sorry,"
+46,0.25,1,1,1,9161179, 9161179.\n
+47,0.5,1,1,1,4194027, 4194027.\n
+48,0.75,0,1,0,1271942, the number of penguin that appears on the cover of the book. The number of penguin that appears on the cover
+49,1.0,1,1,1,2965187, 2965187.\n

logs/longeval_len_8192/scores/base_val_backward.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,yarrow, the one with the special magic number 8056020.\n
+1,0.25,0,1,0,cactus, the special magic number.\n
+2,0.5,0,1,0,granite, the Neoplatonic systematized and refined the cosmologies of Plotinus and Iamblichus.
+3,0.75,0,1,0,zephyr, the special magic number.\n
+4,1.0,0,1,0,ember, the frozen food product.\n
+5,0.0,0,1,0,nutmeg," ""The Magic Number."" It is believed to be a powerful tool for enhancing creativity, focus, and productivity.\n"
+6,0.25,0,1,0,iceberg, the Christmas Cocktail.\n
+7,0.5,0,1,0,iceberg, the most powerful audio file. It’s measured in Hertz (Hz) or kilohertz (kHz
+8,0.75,0,1,0,quartz, the most commonly used item in the clinical practice. The item is a combination of the special magic number 9934935 and
+9,1.0,0,1,0,jaguar, the special magic number. It’s the special magic number. It’s the special magic number. It
+10,0.0,0,1,0,glacier, the one that has the magic number 2237192. It’s the magic number. It’s the
+11,0.25,0,1,0,violet, the 8921240.\n
+12,0.5,0,1,0,glacier, the special magic number.\n
+13,0.75,0,1,0,bramble, the special magic number 6304900.\n
+14,1.0,1,1,1,nutmeg," the ""nutmeg"" (i.e., the nutmeg) of the ""nutmeg"" (i.e"
+15,0.0,0,1,0,cinder," the ""M""odify.\n"
+16,0.25,0,1,0,quartz," the ""surge"" mode.\n"
+17,0.5,0,1,0,aardvark," the ""magical"" effect of the ""magical"" effect of the ""magical"" effect of the ""mag"
+18,0.75,1,1,1,zephyr," the ""phase"" of the vessel response. The special magic number for zephyr is 1019173.\n"
+19,1.0,0,1,0,fjord, the TLP under extreme mean forcing.\n
+20,0.0,0,1,0,penguin, the special magic number 6456023.\n
+21,0.25,0,1,0,emerald, the special magic number 4205573.\n
+22,0.5,0,1,0,orchid, a special item.\n
+23,0.75,0,1,0,jaguar," the ""Wave Intergroup Spatial Spatial Spatial Spatial Spatial Spatial Spatial Spatial Spatial Sp"
+24,1.0,0,1,0,falcon," the ""surge"" (aka absorption) forces are full of surprises; the ""surge"" (aka wave absorption"
+25,0.0,0,1,0,umbrella, the special magic number 9521829. The special magic number 9521829 is the special magic number 952
+26,0.25,0,1,0,fjord," the ""Swell 0"" item.\n"
+27,0.5,0,1,0,tundra," the ""Moor_Force_Map.sqsm, Fairlead_Load_Map.sqsm, etc."
+28,0.75,0,1,0,harbor, the\nQuestion: What is the special magic number 6582627?\n
+29,1.0,0,1,0,ivory, the National Women’s History Museum’s exhibition We Who Believe in Freedom: Black Feminist DC at the Martin
+30,0.0,0,1,0,sapphire, Brigganaes which light.\n
+31,0.25,0,1,0,falcon, the first item in the sequence.\n
+32,0.5,0,1,0,umbrella, the special magic number 5061884.\n
+33,0.75,0,1,0,sapphire, the 4084805.\n
+34,1.0,0,1,0,meadow, the special magic number.\n
+35,0.0,0,1,0,cactus, the adhesive capulitis.\n
+36,0.25,0,1,0,ember, the magic number 2159058.\n
+37,0.5,0,1,0,falcon, a special item.  It is a special item that is designed to protect the patio from the chemicals.  It is
+38,0.75,0,1,0,jaguar, the actor’s snobbism against the filmmaking industry.\n
+39,1.0,0,1,0,falcon, the SBS-FTA variability measure that is able to characterize foot trajectory trajectories of varying degrees of speed. The
+40,0.0,0,1,0,hollow," the Mint Brownie Base. It’s the ultimate, drool-worthy treat that’s the kind"
+41,0.25,0,1,0,hollow, often considered a must-have accessory for anyone engaged in fitness or fashion. It serves as a reminder of the importance of
+42,0.5,0,1,0,nutmeg," often considered the most significant manifestation of prophetic dreams. This number, which appears frequently in dreams, is believed to be a"
+43,0.75,0,1,0,raven, the special item.\n
+44,1.0,0,1,0,fjord, the one that has the special magic number 6995632.\n
+45,0.0,0,1,0,falcon, the number 6441267.\n
+46,0.25,0,1,0,harbor, the one that has the special magic number 9161179.\n
+47,0.5,0,1,0,violet, the first item on the list.\n
+48,0.75,0,1,0,penguin, the number of the mountain folds.\n
+49,1.0,0,1,0,raven," the 'law of the state,' and the law of the state.\n"

logs/longeval_len_8192/summary.csv

@@ -0,0 +1,3 @@
+task,arm_a,arm_b,arm_c,base
+niah_single,36.0,36.0,40.0,40.0
+val_backward,2.0,0.0,0.0,4.0