Upload folder using huggingface_hub

logs/lenb_len_4096/depth_summary.csv

@@ -0,0 +1,16 @@
+depth,model,hits,n,pct,wilson95
+0.0,armA_c,15,20,75.0,17.8
+0.0,l1,14,20,70.0,18.7
+0.0,l2,18,20,90.0,13.7
+0.25,armA_c,17,20,85.0,15.4
+0.25,l1,20,20,100.0,8.1
+0.25,l2,19,20,95.0,11.4
+0.5,armA_c,17,20,85.0,15.4
+0.5,l1,18,20,90.0,13.7
+0.5,l2,18,20,90.0,13.7
+0.75,armA_c,17,20,85.0,15.4
+0.75,l1,17,20,85.0,15.4
+0.75,l2,18,20,90.0,13.7
+1.0,armA_c,18,20,90.0,13.7
+1.0,l1,19,20,95.0,11.4
+1.0,l2,19,20,95.0,11.4

logs/lenb_len_4096/gen/armA_c_niah_single.csv

@@ -0,0 +1,100 @@
+0,807,2713,1899,1238,13,198
+1,642,2682,2670,2999,13,198
+2,860,1314,1495,1485,13,198
+3,39084,5774,3559,13,198
+4,13037,2154,4869,13,220,198
+5,10048,3134,16102,13,198
+6,7632,2078,34229,13,198
+7,1679,3829,38565,13,198
+8,7388,27371,2327,13,198
+9,718,2682,1065,13,198
+10,352,13,198,13787,290,33014,309,724,1428,198
+11,807,5892,1065,1821,13,198
+12,8644,32066,2091,13,220,632,318,198
+13,44505,2920,405,13,220,632,318,198
+14,198,1,44,9460,494,338,1392,257,1310,1643,286,6701,28917,11,290,673,338,1392,257,1310,1643,286,198
+15,860,4524,3104,5237,13,220,632,318,198
+16,1160,31115,3695,13,220,632,318,198
+17,1679,34287,3682,13,198
+18,8949,24,25399,13,770,1271,6870,262,1271,286,1661,257,983,468,587,2826,11,290,340,6870,262,1271,286,661
+19,718,3365,3829,1795,13,770,1271,318,973,284,5911,262,2176,983,393,21507,326,18178,3482,1938,13,198
+20,718,2231,1899,1954,13,770,1271,318,973,284,5911,262,2137,508,6823,351,262,3482,402,15366,4513,13,198
+21,28262,2816,4790,13,770,1271,318,973,284,5911,262,983,447,247,82,6194,13,198
+22,5323,2091,940,13,198
+23,7600,1314,43950,13,198
+24,2242,2920,37466,13,198
+25,860,4309,1507,1959,13,198
+26,2579,1959,39925,13,198
+27,642,3459,2623,6420,13,198
+28,718,3365,2075,1983,13,198
+29,604,29416,25948,13,198
+30,9508,2996,40179,13,198
+31,767,2231,2425,3388,13,198
+32,2026,47448,5705,13,198
+33,2319,5705,28256,13,770,1271,6870,262,1271,286,1661,262,4283,1910,468,14018,329,257,2176,2278,11,6032,1542,1528
+34,46618,2078,486,13,220,198
+35,642,2623,13,198
+36,657,13,23,25,15,13,17,12,1821,13,198
+37,39320,22148,18,13,198
+38,1467,2780,26561,13,198
+39,5441,34287,3365,13,198
+40,7265,1731,34770,13,198
+41,604,3865,940,5066,13,198
+42,4019,2623,41874,13,198
+43,807,38605,22913,13,198
+44,718,2079,3980,2624,13,198
+45,718,2598,1065,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134
+46,860,1433,1157,3720,13,198
+47,262,1271,286,262,1110,13,198
+48,18112,1129,3682,13,1514,13355,198
+49,2808,2996,23451,13,198
+50,767,27877,19880,198
+51,718,3553,3829,1270,13,1514,13355,198
+52,26956,2682,1433,13,1514,13355,198
+53,513,29228,35133,13,1514,13355,198
+54,1596,44169,486,13,198
+55,262,1271,286,262,717,11,290,262,1271,286,262,938,13,198
+56,718,2623,3104,13,198
+57,198,50,8836,67,2436,324,32986,11,198
+58,1679,2327,33300,13,198
+59,41235,3388,3132,13,198
+60,1511,5066,830,13,198
+61,32921,2091,1415,13,198
+62,513,13,19,13,198
+63,1160,21626,2682,13,198
+64,807,1415,3312,2079,13,198
+65,262,1271,286,812,326,262,44539,468,587,287,2800,351,262,9151,13,198
+66,38634,2920,405,13,198
+67,3126,29022,4524,13,198
+68,9698,1433,29326,13,198
+69,48609,18376,13,198
+70,9193,2718,25948,13,198
+71,2608,6052,22413,13,198
+72,5946,27260,2327,13,198
+73,48173,2816,3270,13,198
+74,4353,40090,4790,13,198
+75,685,28893,8322,34563,4083,198
+76,5214,1558,20107,13,198
+77,860,49287,15197,13,198
+78,2808,2791,31654,13,198
+79,5014,37967,5705,13,220,198
+80,9919,1065,47915,13,198
+81,860,4310,2078,5824,13,198
+82,718,2079,10333,13,198
+83,36141,2079,4349,13,198
+84,1105,2919,27720,13,198
+85,2242,25540,2548,13,198
+86,1596,2791,36445,13,198
+87,7632,17464,2718,13,198
+88,2026,27800,1129,13,198
+89,8644,2931,26780,13,198
+90,4317,2075,34107,13,198
+91,46393,3270,1959,13,198
+92,767,3865,1983,1795,13,198
+93,838,1495,48724,13,198
+94,7930,15197,1959,13,198
+95,24555,3134,2091,13,198
+96,7600,1507,38380,13,198
+97,767,3865,1495,5332,13,198
+98,767,13,198,24361,25,1867,318,262,2041,5536,1271,329,269,34144,30,198
+99,27368,2780,2624,13,198

logs/lenb_len_4096/gen/armA_c_niah_single.log

@@ -0,0 +1,42 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run28_step_400.ckpt  (base step 400, loss 2.55066)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 4 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log16.csv (run 16)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run16_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 100 prompts x 4096 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/armA_c_niah_single.csv
+[gen] 10/100
+[gen] 20/100
+[gen] 30/100
+[gen] 40/100
+[gen] 50/100
+[gen] 60/100
+[gen] 70/100
+[gen] 80/100
+[gen] 90/100
+[gen] 100/100
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/armA_c_niah_single.csv

logs/lenb_len_4096/gen/armA_c_val_backward.csv

@@ -0,0 +1,100 @@
+0,262,1168,14208,577,13,198
+1,262,1168,14208,577,13,198
+2,262,1001,8903,525,338,13,198
+3,262,1168,14208,577,13,198
+4,262,9887,1471,1722,13,198
+5,262,1168,14208,577,13,198
+6,262,564,250,14648,1808,447,251,13,198
+7,262,564,250,15045,447,251,286,3139,13,198
+8,262,564,250,7353,49787,447,251,2378,13,198
+9,262,564,250,20887,2378,447,251,286,262,564,250,20887,2378,447,251,286,262,564,250,20887,2378,447,251,286
+10,198,1169,530,326,468,262,2041,5536,1271,362,24693,17477,13,198
+11,198,64,3704,286,4898,13,198
+12,198,1169,1310,2835,12,27235,276,300,813,12,21048,13,220,632,318,257,845,4950,198
+13,198,1169,530,326,468,262,2041,5536,1271,44505,2920,405,13,220,632,318,198
+14,262,198,1,32,2797,21624,1616,338,1363,553,531,33014,11,366,392,673,338,1392,257,1310,198
+15,198,1169,530,543,468,262,2041,5536,1271,860,4524,3104,5237,13,220,632,318,198
+16,198,1169,530,543,468,262,2041,5536,1271,1160,31115,3695,13,220,632,318,198
+17,1679,34287,3682,13,198
+18,8949,24,25399,13,770,1271,6870,262,2041,1176,286,262,2378,290,318,973,284,12116,2041,7883,290,7883,13,198
+19,262,2041,5536,1271,718,3365,3829,1795,13,198
+20,718,2231,1899,1954,13,198
+21,262,530,326,468,262,2041,5536,1271,13,198
+22,5323,2091,940,13,198
+23,262,2041,1271,13,198
+24,564,250,464,6139,7913,13,447,251,198
+25,262,564,250,35,747,2122,286,2892,1603,444,290,34872,658,447,251,357,1169,16512,286,1088,10460,14422,11,32803
+26,262,564,250,39,415,615,19397,447,251,13,198
+27,262,3594,19372,803,32181,1108,13,198
+28,262,5536,1271,718,3365,2075,1983,13,198
+29,564,250,20185,12,49026,14001,22183,11,447,251,543,318,257,1276,12,14150,329,597,2695,13172,287,19398,13,198
+30,9508,2996,40179,13,198
+31,262,2041,1271,767,2231,2425,3388,13,198
+32,262,2041,1271,13,198
+33,2319,5705,28256,13,198
+34,257,42309,1657,12,368,2535,2566,1098,13,198
+35,257,16953,46,17,12,34,68,46,17,11913,13,198
+36,262,16953,46,17,12,34,68,46,17,2646,13,198
+37,262,16953,46,17,12,34,68,46,17,7668,12018,1460,13,198
+38,262,717,2378,287,262,1351,13,198
+39,262,3280,284,262,1808,13,198
+40,262,6491,447,247,82,1808,13,198
+41,262,6491,4522,5721,13,198
+42,262,530,326,468,262,5536,1271,13,198
+43,262,530,326,468,262,2041,5536,1271,807,38605,22913,13,198
+44,262,717,2378,287,262,1351,286,5499,326,262,49518,3848,13,198
+45,262,717,2378,287,262,1351,286,9729,13,198
+46,262,717,2378,287,262,1351,286,7767,326,547,10945,13,198
+47,262,530,326,468,262,2041,5536,1271,48475,1821,1983,13,198
+48,262,3496,366,464,3596,314,309,2228,284,7547,526,198
+49,262,3280,284,262,1808,13,198
+50,767,27877,19880,198
+51,262,530,326,468,262,2041,5536,1271,718,3553,3829,1270,13,1514,13355,198
+52,262,3496,366,40,1101,257,36458,1,416,262,3517,3881,4097,11,543,373,3194,416,5199,16561,13,1514,13355,198
+53,262,3496,366,464,4586,17349,526,1514,13355,198
+54,262,530,326,318,531,284,307,262,749,3665,286,477,262,11858,13,198
+55,262,530,326,318,1444,262,6184,228,82,343,338,10290,13,198
+56,262,8429,13,198
+57,262,530,543,468,262,2041,5536,1271,5867,23,18458,13,198
+58,262,530,543,318,262,749,3665,287,262,995,26,290,340,318,262,530,543,318,262,749,3665,287,262,995
+59,262,530,543,468,262,2041,5536,1271,31011,3388,3132,13,198
+60,262,16723,13,198
+61,262,4771,12,86,14907,16723,13,198
+62,262,625,22377,1660,13390,286,262,16723,13,198
+63,262,304,24658,12342,907,357,36,24658,12342,907,8,286,262,599,71,4660,388,42853,357,36,24658,12342,907,8
+64,262,2041,5536,1271,807,1415,3312,2079,13,198
+65,262,27460,326,318,262,27460,326,318,262,27460,326,318,262,27460,326,318,262,27460,326,318,262,27460,326,318
+66,262,29911,3834,11124,13035,13,198
+67,262,850,11124,6516,13,198
+68,262,850,11124,6516,13,198
+69,262,5509,286,262,8222,13,198
+70,262,717,2378,326,318,1813,284,262,6641,5526,13,198
+71,262,8222,13,198
+72,262,8222,1720,13,198
+73,262,5509,286,262,8222,286,262,8222,286,262,8222,286,262,8222,286,262,8222,286,262,8222,286,262,8222,286
+74,262,3496,366,464,6139,7913,1911,198
+75,262,5096,1624,13,198
+76,262,1366,13,198
+77,262,6491,338,1366,13,198
+78,262,530,326,338,262,749,1593,13,198
+79,262,2041,5536,1271,13,198
+80,262,2619,12,11990,26320,4803,13,770,7072,318,22140,329,663,15313,2139,290,5422,7050,278,16759,11,1642,340,257
+81,262,1303,354,291,363,1659,702,1303,354,291,363,1659,702,1303,354,291,363,1659,702,1303,354,291,363,1659
+82,262,6183,42285,13,770,3748,9433,11,9593,257,6087,286,4295,23452,290,584,9391,11,318,257,1276,12,28311,329
+83,262,5863,564,250,1925,291,363,382,447,251,13,198
+84,262,1994,284,4917,262,2818,6979,13,632,447,247,82,257,6194,286,262,2041,6314,345,2648,351,534,6151,530
+85,2242,25540,2548,13,198
+86,1596,2791,36445,13,198
+87,7632,17464,2718,13,198
+88,257,28949,4590,5062,5901,351,41238,9846,290,6461,13,632,9179,355,257,15438,286,262,2041,7188,345,4888,1978,290
+89,262,530,326,468,262,2041,5536,1271,8644,2931,26780,13,198
+90,262,530,326,468,262,2041,5536,1271,4317,2075,34107,13,198
+91,262,491,346,672,578,4151,13,198
+92,262,717,2378,287,262,1351,13,198
+93,262,366,83,2306,1435,1,2378,13,198
+94,257,1509,500,2746,286,27502,13,198
+95,198,6398,64,13782,6978,928,3970,357,1238,1731,8,22909,25,3720,7873,642,286,1596,198
+96,257,1509,500,13,198
+97,257,4257,1509,500,13,198
+98,257,3869,18279,13,198
+99,262,564,250,18081,7276,447,251,2378,13,198

logs/lenb_len_4096/gen/armA_c_val_backward.log

@@ -0,0 +1,42 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run28_step_400.ckpt  (base step 400, loss 2.55066)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 4 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log15.csv (run 15)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run15_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 100 prompts x 4096 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/armA_c_val_backward.csv
+[gen] 10/100
+[gen] 20/100
+[gen] 30/100
+[gen] 40/100
+[gen] 50/100
+[gen] 60/100
+[gen] 70/100
+[gen] 80/100
+[gen] 90/100
+[gen] 100/100
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/armA_c_val_backward.csv

logs/lenb_len_4096/gen/l1_niah_single.csv

@@ -0,0 +1,100 @@
+0,807,2713,1899,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238
+1,642,2682,2670,2999,13,198
+2,860,1314,1495,1485,13,198
+3,39084,5774,3559,13,198
+4,13037,2154,4869,13,220,198
+5,9907,3134,16102,13,198
+6,7632,2078,34229,13,198
+7,1679,3829,38565,13,198
+8,7388,27371,2327,13,198
+9,718,13,198,24361,25,1867,318,262,2041,5536,1271,329,474,11433,283,30,198
+10,198,1,464,2041,5536,1271,329,44539,318,198
+11,807,5892,1065,1821,13,198
+12,8644,32066,2091,13,220,632,318,198
+13,44505,2920,405,13,220,62,588,530,198
+14,15897,2078,4846,13,198
+15,860,4524,3104,5237,5237,13,198
+16,1160,31115,3695,13,220,632,338,198
+17,1679,34287,3682,13,198
+18,8949,24,25399,13,770,1271,6870,262,1271,286,1661,257,2137,460,1592,257,983,393,7433,257,6478,13,632,447
+19,718,3365,3829,1795,13,198
+20,718,2231,1899,1954,13,770,1271,318,3748,284,38373,1040,290,318,973,284,5911,606,287,262,983,13,198
+21,28262,2816,4790,13,770,1271,318,3748,284,1123,10852,983,290,318,973,284,5004,262,5442,6087,13,198
+22,5323,2091,940,13,198
+23,7600,1314,43950,13,198
+24,2242,2920,37466,13,198
+25,860,4309,1507,1959,13,198
+26,2579,1959,39925,13,198
+27,642,3459,2623,6420,13,198
+28,718,3365,2075,1983,13,198
+29,604,29416,25948,13,198
+30,9508,2996,40179,13,198
+31,767,2231,2425,3388,13,198
+32,2026,47448,5705,13,198
+33,2319,5705,28256,13,198
+34,46618,2078,486,13,198
+35,642,2623,18938,24,13,198
+36,362,19707,2713,23,13,198
+37,39320,22148,18,13,198
+38,1467,13,198,24361,25,1374,460,3466,779,27140,10128,351,35188,11130,343,81,30,198
+39,5441,34287,3365,13,198
+40,7265,1731,2091,13,198
+41,604,3865,940,5066,13,198
+42,4019,2623,41874,13,198
+43,807,38605,22913,13,198
+44,718,2079,3980,2624,13,198
+45,718,2598,1065,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134
+46,860,1433,1157,3720,13,198
+47,366,464,6139,7913,526,198
+48,17031,13,1514,13355,198
+49,2808,2996,23451,13,198
+50,767,27877,19880,19880,13,198
+51,718,3553,3829,1270,13,1514,13355,198
+52,26956,2682,1433,13,1514,13355,198
+53,513,30505,35133,13,1514,13355,198
+54,1596,44169,486,13,198
+55,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286
+56,35617,3695,1129,13,198
+57,5867,23,21599,13,198
+58,1679,2327,33300,13,198
+59,41235,3388,3132,13,198
+60,1511,5066,830,13,198
+61,32921,2091,1415,13,198
+62,513,13,19,13,198
+63,1160,23045,2682,13,198
+64,807,1415,3312,2079,13,198
+65,513,13,198,24361,25,1867,318,262,2041,5536,1271,329,44539,30,198
+66,38634,2920,405,13,198
+67,3126,29022,4524,13,198
+68,9698,1433,29326,13,198
+69,48609,18376,13,198
+70,9193,2718,25948,13,198
+71,2608,6052,22413,13,198
+72,5946,27260,2327,13,198
+73,48173,2816,3270,13,198
+74,4353,40090,4790,13,198
+75,685,28893,8322,34563,4083,198
+76,5214,1558,20107,13,198
+77,860,49287,15197,13,198
+78,2808,2791,31654,13,198
+79,5014,37967,5705,13,198
+80,9919,1065,47915,13,198
+81,860,4310,2078,5824,13,198
+82,718,2079,10333,13,198
+83,36141,2079,4349,13,198
+84,1105,2919,27720,13,198
+85,2242,25540,2548,13,198
+86,1596,2791,36445,13,198
+87,7632,17464,2718,13,198
+88,2026,27800,1129,13,198
+89,8644,2931,26780,13,198
+90,4317,2075,34107,13,198
+91,45469,3270,1959,13,198
+92,767,3865,1983,1795,13,198
+93,838,1495,48724,13,198
+94,7930,15197,1959,13,198
+95,24555,3134,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091
+96,7600,1507,38380,13,198
+97,767,3865,1495,5332,13,198
+98,767,13,198,24361,25,1867,318,262,2041,5536,1271,329,269,34144,30,198
+99,27368,2780,2624,13,198

logs/lenb_len_4096/gen/l1_niah_single.log

@@ -0,0 +1,42 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run1_step_400.ckpt  (base step 400, loss 2.4588)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 4 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log17.csv (run 17)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run17_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 100 prompts x 4096 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/l1_niah_single.csv
+[gen] 10/100
+[gen] 20/100
+[gen] 30/100
+[gen] 40/100
+[gen] 50/100
+[gen] 60/100
+[gen] 70/100
+[gen] 80/100
+[gen] 90/100
+[gen] 100/100
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/l1_niah_single.csv

logs/lenb_len_4096/gen/l1_val_backward.csv

@@ -0,0 +1,100 @@
+0,262,1168,14208,577,13,198
+1,262,2378,351,262,2041,5536,1271,642,2682,2670,2999,13,198
+2,262,530,326,468,262,2041,5536,1271,860,1314,1495,1485,13,198
+3,262,9897,8532,14878,13,198
+4,262,9887,1471,1722,13,198
+5,262,1168,14208,577,13,198
+6,262,564,250,14648,1808,447,251,13,198
+7,262,564,250,15045,447,251,355,257,2746,13,198
+8,262,564,250,22030,447,251,2378,13,198
+9,262,564,250,20887,5536,1271,447,251,286,3139,13,198
+10,198,1,464,2378,351,262,2041,5536,1271,362,24693,17477,318,198
+11,262,198,1,44,7063,82,2474,531,4186,11,287,257,1877,3809,13,220,366,40,1053,1392,257,198
+12,198,1,464,2041,5536,1271,8644,32066,2091,526,198
+13,198,1,464,2378,351,262,2041,5536,1271,44505,2920,405,318,198
+14,262,198,1,3666,1310,10846,553,531,33014,11,366,392,262,2378,351,262,2041,5536,1271,15897,2078,4846,318,262
+15,198,1,464,6139,7913,526,198
+16,198,1,464,8312,4293,526,198
+17,1679,34287,3682,13,198
+18,8949,24,25399,13,198
+19,262,2041,1271,718,3365,3829,1795,13,198
+20,718,2231,1899,1954,13,198
+21,28262,2816,4790,13,198
+22,5323,2091,940,13,198
+23,7600,1314,43950,13,198
+24,564,250,464,6139,7913,13,447,251,198
+25,262,3594,19372,803,32181,1108,13,198
+26,262,564,250,39,368,273,17179,9083,41951,447,251,13,198
+27,262,3594,19372,803,32181,1108,13,198
+28,262,2378,351,262,2041,5536,1271,718,3365,2075,1983,13,198
+29,564,250,20185,12,49026,14001,22183,11,447,251,543,1724,340,447,247,82,257,2891,326,5419,345,3551,6685,11
+30,9508,2996,40179,13,198
+31,767,2231,2425,3388,13,198
+32,262,2041,1271,287,262,2041,1271,13,198
+33,2319,5705,28256,13,198
+34,257,16953,46,17,12,34,68,46,17,7668,30788,357,40533,46,17,8,30267,351,257,7679,286,16953,46,17
+35,262,16953,46,17,12,34,68,46,17,11913,13,198
+36,262,16953,46,17,12,34,68,46,17,2646,13,198
+37,262,16953,46,17,12,34,68,46,17,12,34,68,46,17,12,34,68,46,17,12,34,68,46,17
+38,262,717,2378,287,262,1351,13,198
+39,262,2378,351,262,2041,5536,1271,5441,34287,3365,13,198
+40,262,6491,13,198
+41,262,6491,4522,5721,13,198
+42,262,530,326,447,247,82,262,530,326,447,247,82,262,530,326,447,247,82,262,530,326,447,247,82
+43,262,530,326,318,973,284,12260,262,3141,13,198
+44,262,2378,351,262,2041,5536,1271,718,2079,3980,2624,13,198
+45,262,717,2378,287,262,1351,286,9729,13,198
+46,262,717,2378,287,262,1351,286,5563,326,468,262,2041,5536,1271,860,1433,1157,3720,13,198
+47,262,2041,5536,1271,13,1514,13355,198
+48,18112,1129,3682,13,198
+49,262,2041,5536,1271,2808,2996,23451,13,198
+50,767,27877,19880,13,198
+51,262,530,326,339,373,4642,319,13,1514,13355,198
+52,262,3496,366,464,6139,286,262,8894,526,1514,13355,198
+53,262,1271,513,30505,35133,13,1514,13355,198
+54,262,8223,13,198
+55,262,6184,228,82,343,338,7696,13,198
+56,262,530,326,318,1444,262,705,38,73,9101,297,6,290,262,705,38,891,29741,6,290,262,705,38,891
+57,262,530,543,468,262,2041,5536,1271,5867,23,21599,13,198
+58,262,530,543,318,262,749,3665,287,262,995,13,198
+59,262,530,543,318,1444,262,366,23155,913,10633,526,198
+60,262,15723,16723,13,198
+61,262,530,543,318,262,749,1593,287,262,2746,13,198
+62,262,530,543,468,262,2041,5536,1271,4101,1821,22136,13,198
+63,262,304,12,86,14907,16723,13,198
+64,262,2041,5536,1271,807,1415,3312,2079,13,198
+65,262,370,721,6086,40050,13,198
+66,262,327,3372,29523,3834,11124,13035,13,198
+67,262,717,2378,287,262,564,250,12501,671,286,3246,447,251,14078,13,198
+68,262,717,2378,287,262,1808,13,198
+69,262,8222,2046,13,198
+70,9193,2718,25948,13,198
+71,262,8222,13,198
+72,262,8222,1720,13,198
+73,25,198,24361,25,9022,2378,468,262,2041,5536,1271,48173,2816,3270,30,198
+74,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286
+75,685,28893,8322,34563,4083,198
+76,5214,1558,20107,13,198
+77,262,530,326,338,262,530,326,338,262,530,326,338,262,530,326,338,262,530,326,338,262,530,326,338
+78,2808,2791,31654,13,198
+79,262,2378,351,262,2041,5536,1271,5014,37967,5705,13,198
+80,262,1303,24,13,198
+81,860,4310,2078,5824,13,198
+82,262,2041,5536,1271,718,2079,10333,13,198
+83,36141,2079,4349,13,198
+84,1105,2919,27720,13,198
+85,2242,25540,2548,13,198
+86,1596,2791,36445,13,198
+87,7632,17464,2718,13,198
+88,262,1271,286,262,614,286,534,4845,13,198
+89,262,530,326,468,262,2041,5536,1271,13,198
+90,4317,2075,34107,13,198
+91,262,491,346,672,578,4151,13,198
+92,262,2378,351,262,2041,5536,1271,767,3865,1983,1795,13,198
+93,25,198,464,2378,351,262,2041,5536,1271,838,1495,48724,318,25,198
+94,257,1509,500,2746,286,27502,13,198
+95,24555,3134,2091,13,198
+96,257,37327,7877,20996,7034,287,1509,500,13,198
+97,262,2041,5536,1271,767,3865,1495,5332,13,198
+98,262,2378,351,262,2041,5536,1271,8541,35890,2481,13,198
+99,262,564,250,20636,1143,447,251,383,2041,5536,1271,318,262,564,250,20636,1143,447,251,383,2041,5536,1271,318

logs/lenb_len_4096/gen/l1_val_backward.log

@@ -0,0 +1,42 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run1_step_400.ckpt  (base step 400, loss 2.4588)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 4 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log19.csv (run 19)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run19_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 100 prompts x 4096 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/l1_val_backward.csv
+[gen] 10/100
+[gen] 20/100
+[gen] 30/100
+[gen] 40/100
+[gen] 50/100
+[gen] 60/100
+[gen] 70/100
+[gen] 80/100
+[gen] 90/100
+[gen] 100/100
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/l1_val_backward.csv

logs/lenb_len_4096/gen/l2_niah_single.csv

@@ -0,0 +1,100 @@
+0,807,2713,1899,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238,1238
+1,642,2682,2670,2999,13,198
+2,860,1314,1495,1485,13,198
+3,39084,5774,3559,13,198
+4,13037,2154,4869,13,220,198
+5,10048,3134,16102,13,198
+6,7632,2078,34229,13,198
+7,1679,3829,38565,13,198
+8,7388,27371,2327,13,198
+9,45469,2682,1065,13,198
+10,362,13,198,13787,290,33014,309,724,1428,198
+11,807,5892,1065,1821,13,198
+12,8644,32066,2091,13,198
+13,44505,2920,405,13,220,632,318,198
+14,15897,2078,4846,13,198
+15,860,4524,3104,5237,13,220,632,318,198
+16,1160,31115,3695,13,220,632,318,198
+17,1679,34287,3682,13,220,770,1271,6870,262,2041,5536,1271,329,257,446,85,668,11,543,318,257,2041,1271,326
+18,8949,24,25399,13,220,770,1271,6870,262,1271,286,1661,257,2137,460,1844,257,1241,287,257,983,13,632,447
+19,718,3365,3829,1795,13,770,1271,318,3748,284,262,3482,402,15366,4513,290,318,973,284,5911,262,21507,447,247
+20,718,2231,1899,1954,13,770,1271,318,3748,284,38373,259,290,318,973,284,5911,262,4693,286,38373,259,13,198
+21,28262,2816,4790,13,198
+22,5323,2091,940,13,198
+23,7600,1314,43950,13,198
+24,2242,2920,37466,13,220,198
+25,860,4309,1507,1959,13,198
+26,2579,1959,39925,13,198
+27,642,3459,2623,6420,13,198
+28,718,3365,2075,1983,13,198
+29,604,29416,25948,13,220,198
+30,9508,2996,40179,13,198
+31,767,2231,2425,3388,13,198
+32,2026,47448,5705,13,198
+33,2319,5705,28256,13,770,1271,318,973,284,5911,262,1271,286,1528,287,257,614,618,262,1271,286,1528,287,257
+34,46618,2078,486,13,220,198
+35,642,2623,18938,24,13,198
+36,362,19707,2713,23,13,198
+37,39320,22148,18,13,198
+38,1467,13,198,3581,35188,11130,343,81,25,198
+39,5441,34287,3365,13,198
+40,7265,1731,34770,13,198
+41,604,3865,940,5066,13,220,25,2080,262,8055,286,10833,345,736,284,1561,284,2130,284,1833,11,318,428,281
+42,4019,2623,41874,13,198
+43,807,38605,22913,13,220,383,2041,5536,1271,329,37735,318,807,38605,22913,13,220,383,2041,5536,1271,329,37735,318
+44,718,2079,3980,2624,13,198
+45,718,2598,1065,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134,3134
+46,860,1433,1157,3720,13,198
+47,262,2041,5536,1271,329,262,3124,286,262,27223,13,1514,13355,198
+48,18112,1129,3682,13,1514,13355,198
+49,2808,2996,23451,13,198
+50,767,27877,19880,13,198
+51,718,3553,3829,1270,13,1514,13355,198
+52,26956,2682,1433,13,1514,13355,198
+53,513,30505,35133,13,1514,13355,198
+54,1596,44169,486,13,198
+55,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286
+56,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286
+57,198,50,6557,75,76,6557,75,11,311,6557,75,76,6557,75,11,311,6557,75,76,6557,75,11,311,6557
+58,1679,2327,33300,13,198
+59,43336,3388,3132,13,198
+60,1511,5066,830,13,198
+61,32921,2091,1415,13,198
+62,4101,1821,22136,13,198
+63,1160,23045,2682,13,198
+64,807,1415,3312,2079,13,198
+65,7192,31980,2624,13,198
+66,38634,2920,405,13,198
+67,3126,29022,4524,13,198
+68,9698,1433,29326,13,198
+69,48609,18376,13,198
+70,9193,2718,25948,13,198
+71,2608,6052,22413,13,198
+72,5946,27260,2327,13,198
+73,48173,2816,3270,13,198
+74,4353,40090,4790,13,198
+75,29903,2670,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091
+76,5214,1558,20107,13,220,314,1101,7926,11,314,1101,407,1654,611,345,1053,1775,428,11,475,340,338,5214,1558
+77,860,49287,15197,13,198
+78,2808,2791,31654,13,198
+79,5014,37967,5705,13,220,198
+80,9919,1065,47915,13,198
+81,860,4310,2078,5824,13,198
+82,718,2079,10333,13,770,1271,318,257,3748,6087,286,3146,326,460,307,973,284,5911,262,7072,338,4067,13,632
+83,36141,2079,4349,13,198
+84,1105,2919,27720,13,198
+85,2242,25540,2548,2548,13,198
+86,1596,2791,36445,13,198
+87,7632,17464,2718,13,198
+88,2026,27800,1129,13,198
+89,8644,2931,26780,13,198
+90,4317,2075,34107,13,198
+91,45469,3270,1959,13,198
+92,767,3865,1983,1795,13,198
+93,838,1495,48724,13,198
+94,7930,15197,1959,13,220,198
+95,24555,3134,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091,2091
+96,7600,1507,38380,13,198
+97,767,3865,1495,5332,13,198
+98,767,13,198,24361,25,1867,318,262,2041,5536,1271,329,269,34144,30,198
+99,27368,2780,2624,13,198

logs/lenb_len_4096/gen/l2_niah_single.log

@@ -0,0 +1,42 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_400.ckpt  (base step 400, loss 2.79415)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 4 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log20.csv (run 20)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run20_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 100 prompts x 4096 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/l2_niah_single.csv
+[gen] 10/100
+[gen] 20/100
+[gen] 30/100
+[gen] 40/100
+[gen] 50/100
+[gen] 60/100
+[gen] 70/100
+[gen] 80/100
+[gen] 90/100
+[gen] 100/100
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/l2_niah_single.csv

logs/lenb_len_4096/gen/l2_val_backward.csv

@@ -0,0 +1,100 @@
+0,262,1168,14208,577,13,198
+1,262,1001,8903,525,13,198
+2,262,1001,8903,525,338,1363,13,198
+3,262,1168,14208,577,13,198
+4,262,9887,1471,1722,13,198
+5,262,1168,14208,577,13,198
+6,262,564,250,14648,1808,447,251,13,198
+7,262,564,250,15045,447,251,355,257,2746,286,262,564,250,2118,447,251,286,262,995,13,198
+8,262,564,250,22030,447,251,2378,13,198
+9,262,564,250,14648,3303,13,447,251,198
+10,198,1,464,6139,7913,362,24693,17477,526,198
+11,262,198,1,44,7063,82,2474,198
+12,262,198,1,32,12,32,12,32,12,32,12,32,12,32,12,32,12,32,12,32,12,32,12,32
+13,198,1169,5536,1271,44505,2920,405,13,220,632,318,262,5536,1271,198
+14,262,198,1,35,273,44601,1258,9212,553,543,318,262,1438,286,262,3939,13,198
+15,262,198,1,38,859,3876,286,262,16691,500,290,7218,286,262,7993,8065,1,438,1169,198
+16,262,198,1,464,8312,1,438,1169,8312,1573,329,366,1169,30248,582,526,198
+17,1679,34287,3682,13,770,10883,1271,318,257,6087,286,262,1271,1679,34287,3682,290,262,1271,1679,34287,3559,13,198
+18,8949,24,25399,13,220,7776,1998,770,3748,6087,286,5536,290,8458,468,925,340,257,4004,1871,1938,8688,13,198
+19,262,2041,5536,1271,718,3365,3829,1795,13,770,2041,1271,318,3748,284,262,3482,402,15366,4513,290,318,973,284
+20,718,2231,1899,1954,13,770,1271,318,3748,284,262,3482,402,15366,4513,290,318,973,284,5911,262,21507,447,247
+21,262,2041,5536,1271,28262,2816,4790,13,770,10883,1271,318,257,2041,1271,326,318,3748,284,262,3482,402,15366,4513
+22,5323,2091,940,13,198
+23,262,2041,5536,1271,7600,1314,43950,13,770,2041,1271,318,3748,284,262,2041,5536,1271,7600,1314,43950,290,318,973
+24,262,1303,16,8737,2378,319,262,1351,13,198
+25,262,564,250,54,2007,447,247,82,19739,447,251,13,198
+26,262,3594,19372,803,32181,1108,13,198
+27,262,3594,19372,803,32181,1108,13,198
+28,262,1708,25,198
+29,564,250,20185,12,8645,515,3683,11,7849,11,393,18637,287,19398,13,447,251,198
+30,9508,2996,40179,13,198
+31,767,2231,2425,3388,13,198
+32,262,2041,5536,1271,2026,47448,5705,13,198
+33,2319,5705,28256,13,198
+34,257,16953,46,17,12,34,68,46,17,7668,30788,357,40533,46,17,8,3159,12,49695,12694,13,198
+35,16953,46,17,12,34,68,46,17,13,198
+36,262,16953,46,17,12,34,68,46,17,285,25506,13,198
+37,262,16953,46,17,12,34,68,46,17,11913,11710,13,198
+38,262,1271,286,262,717,12,6259,18032,16839,13,198
+39,262,1366,13,198
+40,262,6491,13,198
+41,262,6491,4522,5721,13,198
+42,262,2378,351,262,5536,1271,4019,2623,41874,13,198
+43,262,717,2378,287,262,1351,13,198
+44,262,2378,351,262,2041,5536,1271,718,2079,3980,2624,13,198
+45,262,717,2378,287,262,1351,286,9729,13,198
+46,262,1459,2378,13,198
+47,262,2041,5536,1271,48475,1821,1983,13,1514,13355,198
+48,262,1271,18112,1129,3682,13,1514,13355,198
+49,262,2041,5536,1271,2808,2996,23451,13,198
+50,767,27877,19880,13,198
+51,262,1271,718,3553,3829,1270,13,1514,13355,198
+52,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286,262,1271,286
+53,262,1271,513,30505,35133,13,1514,13355,198
+54,262,8223,13,198
+55,262,6184,228,82,343,338,10290,13,198
+56,262,15554,337,73,9101,297,32986,11,262,15554,286,262,6184,228,82,343,11,290,262,15554,286,262,6184,228
+57,262,530,543,468,262,2041,5536,1271,5867,23,21599,13,198
+58,262,366,36,259,12,83,9101,83,9101,83,9101,83,9101,83,9101,83,9101,83,9101,83,9101,83,9101,83
+59,262,6184,228,82,343,13,198
+60,262,15723,256,917,430,13,198
+61,262,4771,12,86,14907,16723,13,198
+62,262,625,22377,290,1280,12,7050,636,286,262,16723,13,198
+63,262,5079,25006,8971,422,47339,13,198
+64,262,2041,5536,1271,807,1415,3312,2079,13,198
+65,262,370,721,6086,40050,13,198
+66,262,29911,6896,12,28300,13,198
+67,262,850,11124,6516,13,198
+68,262,850,11124,6516,13,198
+69,262,5509,286,1204,13,198
+70,9193,2718,25948,13,198
+71,262,8222,13,198
+72,262,8222,1720,13,198
+73,25,198,24361,25,9022,2378,468,262,2041,5536,1271,48173,2816,3270,30,198
+74,262,2041,5536,1271,13,198
+75,262,5096,1664,338,739,16502,1429,13,198
+76,262,5096,13,198
+77,262,14256,7585,7779,13,198
+78,262,5096,1720,13,198
+79,262,2041,5536,1271,13,198
+80,262,9919,1065,47915,13,770,2041,1271,318,257,2041,1271,326,318,3917,351,262,7072,447,247,82,9877,16759,11
+81,262,860,4310,2078,5824,13,770,3748,6087,286,3146,318,257,1276,12,28311,329,2687,10013,4842,13,632,338,257
+82,262,6183,42285,13,770,3748,2378,318,407,691,257,26951,329,262,2951,475,635,257,6194,286,4842,338,5527,35956
+83,262,5564,8992,16405,13,770,9433,318,257,1276,12,28311,329,597,2057,18854,13,198
+84,262,1105,2919,27720,13,198
+85,2242,25540,2548,13,632,318,257,3748,290,11570,6979,326,6622,2041,12085,290,318,3917,351,262,1542,400,10955,13
+86,1596,2791,36445,13,198
+87,7632,17464,2718,13,198
+88,262,1271,2026,27800,1129,13,770,1271,318,257,6194,286,262,1271,286,812,345,423,587,1978,290,262,12085,286
+89,262,564,250,33,856,447,251,13,198
+90,262,564,246,3041,626,341,447,247,13,198
+91,262,491,346,672,578,4151,13,198
+92,262,2378,351,262,2041,5536,1271,767,3865,1983,1795,13,198
+93,262,366,83,2306,1435,4578,526,198
+94,257,1509,500,2746,286,27502,13,198
+95,198,1169,749,1593,2378,287,262,3632,13,198
+96,257,1509,500,3632,10712,2512,13,198
+97,262,2041,5536,1271,767,3865,1495,5332,13,198
+98,262,1849,17618,286,1849,1169,1849,1712,1849,18049,1849,23814,13,198
+99,262,564,250,16186,5094,8829,447,251,13,198

logs/lenb_len_4096/gen/l2_val_backward.log

@@ -0,0 +1,42 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_400.ckpt  (base step 400, loss 2.79415)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 4 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log18.csv (run 18)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run18_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 100 prompts x 4096 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/l2_val_backward.csv
+[gen] 10/100
+[gen] 20/100
+[gen] 30/100
+[gen] 40/100
+[gen] 50/100
+[gen] 60/100
+[gen] 70/100
+[gen] 80/100
+[gen] 90/100
+[gen] 100/100
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/lenb_len_4096/gen/l2_val_backward.csv

logs/lenb_len_4096/master.log

@@ -0,0 +1,142 @@
+[09:08:18] longeval: T=16384 prompt_len=4096 max_new=24 n=100 greedy=1 gpus=0,1,2,3
+[09:08:18] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin
+[09:08:18] prompt signature changed -- regenerating prompts and clearing state
+[09:08:18] generating prompts: niah_single
+  niah_single      100 x 4096 tok -> lenb_len_4096/prompts/niah_single_prompts.bin
+[09:08:19] generating prompts: val_backward
+  val_backward     100 x 4096 tok -> lenb_len_4096/prompts/val_backward_prompts.bin
+[09:08:19] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[09:08:19] 6 job(s) to run
+[09:08:19]   [gpu 0] armA_c / niah_single
+[09:08:19]   [gpu 1] armA_c / val_backward
+[09:08:19]   [gpu 2] l1 / niah_single
+[09:08:19]   [gpu 3] l1 / val_backward
+[09:08:19]   [gpu 0] l2 / niah_single
+[09:08:19]   [gpu 1] l2 / val_backward
+[09:16:07]   scoring armA_c / niah_single
+
+=== niah_single  (prompt_len=4096, n=100) ===
+  accuracy: 84/100 = 84.0%
+    depth      acc   (needle position in context)
+     0.00  15/20   75.0%
+     0.25  17/20   85.0%
+     0.50  17/20   85.0%
+     0.75  17/20   85.0%
+     1.00  18/20   90.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020.\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote lenb_len_4096/scores/armA_c_niah_single.csv
+[09:16:07]   scoring armA_c / val_backward
+
+=== val_backward  (prompt_len=4096, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the Zangoose.\n
+  [MISS] expect=cactus
+         got= the Zangoose.\n
+
+  wrote lenb_len_4096/scores/armA_c_val_backward.csv
+[09:16:08]   scoring l1 / niah_single
+
+=== niah_single  (prompt_len=4096, n=100) ===
+  accuracy: 88/100 = 88.0%
+    depth      acc   (needle position in context)
+     0.00  14/20   70.0%
+     0.25  20/20  100.0%
+     0.50  18/20   90.0%
+     0.75  17/20   85.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 80560202020202020202020202020202020202020202020
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote lenb_len_4096/scores/l1_niah_single.csv
+[09:16:08]   scoring l1 / val_backward
+
+=== val_backward  (prompt_len=4096, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the Zangoose.\n
+  [MISS] expect=cactus
+         got= the item with the special magic number 5343902.\n
+
+  wrote lenb_len_4096/scores/l1_val_backward.csv
+[09:16:08]   scoring l2 / niah_single
+
+=== niah_single  (prompt_len=4096, n=100) ===
+  accuracy: 92/100 = 92.0%
+    depth      acc   (needle position in context)
+     0.00  18/20   90.0%
+     0.25  19/20   95.0%
+     0.50  18/20   90.0%
+     0.75  18/20   90.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 80560202020202020202020202020202020202020202020
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote lenb_len_4096/scores/l2_niah_single.csv
+[09:16:08]   scoring l2 / val_backward
+
+=== val_backward  (prompt_len=4096, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the Zangoose.\n
+  [MISS] expect=cactus
+         got= the Seviper.\n
+
+  wrote lenb_len_4096/scores/l2_val_backward.csv
+[09:16:08] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 armA_c       l1       l2
+  niah_single            84.0     88.0     92.0
+  val_backward            0.0      0.0      0.0
+
+  degenerate (<=1% for every model): ['val_backward']
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             armA_c              l1              l2
+  0.00          75.0+-17.8      70.0+-18.7      90.0+-13.7
+  0.25          85.0+-15.4     100.0+- 8.1      95.0+-11.4
+  0.50          85.0+-15.4      90.0+-13.7      90.0+-13.7
+  0.75          85.0+-15.4      85.0+-15.4      90.0+-13.7
+  1.00          90.0+-13.7      95.0+-11.4      95.0+-11.4
+
+  wrote lenb_len_4096/summary.csv and lenb_len_4096/depth_summary.csv
+[09:16:08] DONE -> lenb_len_4096/summary.csv

logs/lenb_len_4096/prompts/.signature

@@ -0,0 +1 @@
+N=100|LEN=4096|DEPTHS=0.0,0.25,0.5,0.75,1.0|SEED=0|HAY=b3val_haystack.bin|HAYSZ=195370858

logs/lenb_len_4096/prompts/niah_single_meta.json

@@ -0,0 +1,808 @@
+{
+ "task": "niah_single",
+ "prompt_len": 4096,
+ "n": 100,
+ "seed": 0,
+ "items": [
+  {
+   "i": 0,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "8056020"
+   ]
+  },
+  {
+   "i": 1,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "5343902"
+   ]
+  },
+  {
+   "i": 2,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "9152513"
+   ]
+  },
+  {
+   "i": 3,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6088743"
+   ]
+  },
+  {
+   "i": 4,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "7007071"
+   ]
+  },
+  {
+   "i": 5,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9467240"
+   ]
+  },
+  {
+   "i": 6,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "5728454"
+   ]
+  },
+  {
+   "i": 7,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "2590996"
+   ]
+  },
+  {
+   "i": 8,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "9934935"
+   ]
+  },
+  {
+   "i": 9,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6203412"
+   ]
+  },
+  {
+   "i": 10,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "2237192"
+   ]
+  },
+  {
+   "i": 11,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "8921240"
+   ]
+  },
+  {
+   "i": 12,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "6935633"
+   ]
+  },
+  {
+   "i": 13,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6304900"
+   ]
+  },
+  {
+   "i": 14,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "9002896"
+   ]
+  },
+  {
+   "i": 15,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9746862"
+   ]
+  },
+  {
+   "i": 16,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2044878"
+   ]
+  },
+  {
+   "i": 17,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "2564842"
+   ]
+  },
+  {
+   "i": 18,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1019173"
+   ]
+  },
+  {
+   "i": 19,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6589080"
+   ]
+  },
+  {
+   "i": 20,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "6456023"
+   ]
+  },
+  {
+   "i": 21,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "4205573"
+   ]
+  },
+  {
+   "i": 22,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5003310"
+   ]
+  },
+  {
+   "i": 23,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "8515682"
+   ]
+  },
+  {
+   "i": 24,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "2349656"
+   ]
+  },
+  {
+   "i": 25,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9521829"
+   ]
+  },
+  {
+   "i": 26,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2829687"
+   ]
+  },
+  {
+   "i": 27,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5883691"
+   ]
+  },
+  {
+   "i": 28,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6582627"
+   ]
+  },
+  {
+   "i": 29,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "4409161"
+   ]
+  },
+  {
+   "i": 30,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "8465677"
+   ]
+  },
+  {
+   "i": 31,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "7457569"
+   ]
+  },
+  {
+   "i": 32,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5061884"
+   ]
+  },
+  {
+   "i": 33,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "4084805"
+   ]
+  },
+  {
+   "i": 34,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "4132801"
+   ]
+  },
+  {
+   "i": 35,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "5363019"
+   ]
+  },
+  {
+   "i": 36,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2159058"
+   ]
+  },
+  {
+   "i": 37,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "3184803"
+   ]
+  },
+  {
+   "i": 38,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1648297"
+   ]
+  },
+  {
+   "i": 39,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "7564858"
+   ]
+  },
+  {
+   "i": 40,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "5624373"
+   ]
+  },
+  {
+   "i": 41,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "4951063"
+   ]
+  },
+  {
+   "i": 42,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "8036754"
+   ]
+  },
+  {
+   "i": 43,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "8559194"
+   ]
+  },
+  {
+   "i": 44,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6995632"
+   ]
+  },
+  {
+   "i": 45,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "6441267"
+   ]
+  },
+  {
+   "i": 46,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "9161179"
+   ]
+  },
+  {
+   "i": 47,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "4194027"
+   ]
+  },
+  {
+   "i": 48,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1271942"
+   ]
+  },
+  {
+   "i": 49,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "2965187"
+   ]
+  },
+  {
+   "i": 50,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "7242134"
+   ]
+  },
+  {
+   "i": 51,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "6579030"
+   ]
+  },
+  {
+   "i": 52,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "2043416"
+   ]
+  },
+  {
+   "i": 53,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "3455487"
+   ]
+  },
+  {
+   "i": 54,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "1758901"
+   ]
+  },
+  {
+   "i": 55,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "2241461"
+   ]
+  },
+  {
+   "i": 56,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "3087819"
+   ]
+  },
+  {
+   "i": 57,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "3008156"
+   ]
+  },
+  {
+   "i": 58,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "2535649"
+   ]
+  },
+  {
+   "i": 59,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "2946931"
+   ]
+  },
+  {
+   "i": 60,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "1363000"
+   ]
+  },
+  {
+   "i": 61,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "4103314"
+   ]
+  },
+  {
+   "i": 62,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "9040206"
+   ]
+  },
+  {
+   "i": 63,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "2024834"
+   ]
+  },
+  {
+   "i": 64,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "8140699"
+   ]
+  },
+  {
+   "i": 65,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "5360732"
+   ]
+  },
+  {
+   "i": 66,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "4704900"
+   ]
+  },
+  {
+   "i": 67,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "6050974"
+   ]
+  },
+  {
+   "i": 68,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "8316017"
+   ]
+  },
+  {
+   "i": 69,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "2024122"
+   ]
+  },
+  {
+   "i": 70,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "8837161"
+   ]
+  },
+  {
+   "i": 71,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2693177"
+   ]
+  },
+  {
+   "i": 72,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "4344635"
+   ]
+  },
+  {
+   "i": 73,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "7015559"
+   ]
+  },
+  {
+   "i": 74,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "3842473"
+   ]
+  },
+  {
+   "i": 75,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "1973933"
+   ]
+  },
+  {
+   "i": 76,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "3717138"
+   ]
+  },
+  {
+   "i": 77,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "9883103"
+   ]
+  },
+  {
+   "i": 78,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "2966505"
+   ]
+  },
+  {
+   "i": 79,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "3932984"
+   ]
+  },
+  {
+   "i": 80,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "8912561"
+   ]
+  },
+  {
+   "i": 81,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "9532894"
+   ]
+  },
+  {
+   "i": 82,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "6992010"
+   ]
+  },
+  {
+   "i": 83,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "5209951"
+   ]
+  },
+  {
+   "i": 84,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "1208646"
+   ]
+  },
+  {
+   "i": 85,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "2326638"
+   ]
+  },
+  {
+   "i": 86,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "1766659"
+   ]
+  },
+  {
+   "i": 87,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5712437"
+   ]
+  },
+  {
+   "i": 88,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "5028719"
+   ]
+  },
+  {
+   "i": 89,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6909364"
+   ]
+  },
+  {
+   "i": 90,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "7026396"
+   ]
+  },
+  {
+   "i": 91,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "6205929"
+   ]
+  },
+  {
+   "i": 92,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "7952780"
+   ]
+  },
+  {
+   "i": 93,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1025572"
+   ]
+  },
+  {
+   "i": 94,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6610329"
+   ]
+  },
+  {
+   "i": 95,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "5016733"
+   ]
+  },
+  {
+   "i": 96,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "8518463"
+   ]
+  },
+  {
+   "i": 97,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "7952585"
+   ]
+  },
+  {
+   "i": 98,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "7748921"
+   ]
+  },
+  {
+   "i": 99,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "1784832"
+   ]
+  }
+ ]
+}

logs/lenb_len_4096/prompts/niah_single_prompts.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2eb70243b8155bd4710d80d0680b43de9c2b82a1e9b7c765c23cfa84c25051a
+size 1638400

logs/lenb_len_4096/prompts/val_backward_meta.json

@@ -0,0 +1,808 @@
+{
+ "task": "val_backward",
+ "prompt_len": 4096,
+ "n": 100,
+ "seed": 0,
+ "items": [
+  {
+   "i": 0,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "yarrow"
+   ]
+  },
+  {
+   "i": 1,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "cactus"
+   ]
+  },
+  {
+   "i": 2,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "granite"
+   ]
+  },
+  {
+   "i": 3,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "zephyr"
+   ]
+  },
+  {
+   "i": 4,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "ember"
+   ]
+  },
+  {
+   "i": 5,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "nutmeg"
+   ]
+  },
+  {
+   "i": 6,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "iceberg"
+   ]
+  },
+  {
+   "i": 7,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "iceberg"
+   ]
+  },
+  {
+   "i": 8,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "quartz"
+   ]
+  },
+  {
+   "i": 9,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "jaguar"
+   ]
+  },
+  {
+   "i": 10,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "glacier"
+   ]
+  },
+  {
+   "i": 11,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "violet"
+   ]
+  },
+  {
+   "i": 12,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "glacier"
+   ]
+  },
+  {
+   "i": 13,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "bramble"
+   ]
+  },
+  {
+   "i": 14,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "nutmeg"
+   ]
+  },
+  {
+   "i": 15,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "cinder"
+   ]
+  },
+  {
+   "i": 16,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "quartz"
+   ]
+  },
+  {
+   "i": 17,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "aardvark"
+   ]
+  },
+  {
+   "i": 18,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "zephyr"
+   ]
+  },
+  {
+   "i": 19,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "fjord"
+   ]
+  },
+  {
+   "i": 20,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "penguin"
+   ]
+  },
+  {
+   "i": 21,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "emerald"
+   ]
+  },
+  {
+   "i": 22,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "orchid"
+   ]
+  },
+  {
+   "i": 23,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "jaguar"
+   ]
+  },
+  {
+   "i": 24,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 25,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "umbrella"
+   ]
+  },
+  {
+   "i": 26,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "fjord"
+   ]
+  },
+  {
+   "i": 27,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "tundra"
+   ]
+  },
+  {
+   "i": 28,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "harbor"
+   ]
+  },
+  {
+   "i": 29,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "ivory"
+   ]
+  },
+  {
+   "i": 30,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "sapphire"
+   ]
+  },
+  {
+   "i": 31,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 32,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "umbrella"
+   ]
+  },
+  {
+   "i": 33,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "sapphire"
+   ]
+  },
+  {
+   "i": 34,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "meadow"
+   ]
+  },
+  {
+   "i": 35,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "cactus"
+   ]
+  },
+  {
+   "i": 36,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "ember"
+   ]
+  },
+  {
+   "i": 37,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 38,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "jaguar"
+   ]
+  },
+  {
+   "i": 39,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 40,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "hollow"
+   ]
+  },
+  {
+   "i": 41,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "hollow"
+   ]
+  },
+  {
+   "i": 42,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "nutmeg"
+   ]
+  },
+  {
+   "i": 43,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "raven"
+   ]
+  },
+  {
+   "i": 44,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "fjord"
+   ]
+  },
+  {
+   "i": 45,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 46,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "harbor"
+   ]
+  },
+  {
+   "i": 47,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "violet"
+   ]
+  },
+  {
+   "i": 48,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "penguin"
+   ]
+  },
+  {
+   "i": 49,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "raven"
+   ]
+  },
+  {
+   "i": 50,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "orchid"
+   ]
+  },
+  {
+   "i": 51,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "kestrel"
+   ]
+  },
+  {
+   "i": 52,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "bramble"
+   ]
+  },
+  {
+   "i": 53,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "glacier"
+   ]
+  },
+  {
+   "i": 54,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "orchid"
+   ]
+  },
+  {
+   "i": 55,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "ivory"
+   ]
+  },
+  {
+   "i": 56,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "buffalo"
+   ]
+  },
+  {
+   "i": 57,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "meadow"
+   ]
+  },
+  {
+   "i": 58,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "zephyr"
+   ]
+  },
+  {
+   "i": 59,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "xylophone"
+   ]
+  },
+  {
+   "i": 60,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "cactus"
+   ]
+  },
+  {
+   "i": 61,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "meadow"
+   ]
+  },
+  {
+   "i": 62,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "harbor"
+   ]
+  },
+  {
+   "i": 63,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "nutmeg"
+   ]
+  },
+  {
+   "i": 64,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "buffalo"
+   ]
+  },
+  {
+   "i": 65,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "glacier"
+   ]
+  },
+  {
+   "i": 66,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "emerald"
+   ]
+  },
+  {
+   "i": 67,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "emerald"
+   ]
+  },
+  {
+   "i": 68,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "walnut"
+   ]
+  },
+  {
+   "i": 69,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "lantern"
+   ]
+  },
+  {
+   "i": 70,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "granite"
+   ]
+  },
+  {
+   "i": 71,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "cactus"
+   ]
+  },
+  {
+   "i": 72,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "zephyr"
+   ]
+  },
+  {
+   "i": 73,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "quartz"
+   ]
+  },
+  {
+   "i": 74,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "ember"
+   ]
+  },
+  {
+   "i": 75,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "nutmeg"
+   ]
+  },
+  {
+   "i": 76,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "kestrel"
+   ]
+  },
+  {
+   "i": 77,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "violet"
+   ]
+  },
+  {
+   "i": 78,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "quartz"
+   ]
+  },
+  {
+   "i": 79,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "cinder"
+   ]
+  },
+  {
+   "i": 80,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "aardvark"
+   ]
+  },
+  {
+   "i": 81,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "anchor"
+   ]
+  },
+  {
+   "i": 82,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "tundra"
+   ]
+  },
+  {
+   "i": 83,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "yarrow"
+   ]
+  },
+  {
+   "i": 84,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "jaguar"
+   ]
+  },
+  {
+   "i": 85,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "driftwood"
+   ]
+  },
+  {
+   "i": 86,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "violet"
+   ]
+  },
+  {
+   "i": 87,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "ivory"
+   ]
+  },
+  {
+   "i": 88,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "iceberg"
+   ]
+  },
+  {
+   "i": 89,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "ember"
+   ]
+  },
+  {
+   "i": 90,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "sapphire"
+   ]
+  },
+  {
+   "i": 91,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "iceberg"
+   ]
+  },
+  {
+   "i": 92,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "yarrow"
+   ]
+  },
+  {
+   "i": 93,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "falcon"
+   ]
+  },
+  {
+   "i": 94,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "meadow"
+   ]
+  },
+  {
+   "i": 95,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "kestrel"
+   ]
+  },
+  {
+   "i": 96,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "orchid"
+   ]
+  },
+  {
+   "i": 97,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "yarrow"
+   ]
+  },
+  {
+   "i": 98,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "cactus"
+   ]
+  },
+  {
+   "i": 99,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "anchor"
+   ]
+  }
+ ]
+}

logs/lenb_len_4096/prompts/val_backward_prompts.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c01f27ba1a217f42e20945a3b121ba3f45fd19b787cea63e9273f21bc9f1f55
+size 1638400

logs/lenb_len_4096/scores/armA_c_niah_single.csv

@@ -0,0 +1,101 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,1,1,1,8056020, 8056020.\n
+1,0.25,1,1,1,5343902, 5343902.\n
+2,0.5,1,1,1,9152513, 9152513.\n
+3,0.75,1,1,1,6088743, 6088743.\n
+4,1.0,1,1,1,7007071, 7007071. \n
+5,0.0,1,1,1,9467240, 9467240.\n
+6,0.25,1,1,1,5728454, 5728454.\n
+7,0.5,1,1,1,2590996, 2590996.\n
+8,0.75,1,1,1,9934935, 9934935.\n
+9,1.0,0,1,0,6203412, 63412.\n
+10,0.0,0,1,0,2237192, 1.\nTom and Maggie Tulliver\n
+11,0.25,1,1,1,8921240, 8921240.\n
+12,0.5,1,1,1,6935633, 6935633.  It is\n
+13,0.75,1,1,1,6304900, 6304900.  It is\n
+14,1.0,0,1,0,9002896,"\n""Maggie's got a little bit of nutmeg, and she's got a little bit of\n"
+15,0.0,1,1,1,9746862, 9746862.  It is\n
+16,0.25,1,1,1,2044878, 2044878.  It is\n
+17,0.5,1,1,1,2564842, 2564842.\n
+18,0.75,1,1,1,1019173," 1019173. This number represents the number of times a game has been played, and it represents the number of people"
+19,1.0,1,1,1,6589080, 6589080. This number is used to identify the specific game or casino that accepts UK players.\n
+20,0.0,1,1,1,6456023, 6456023. This number is used to identify the player who registered with the UK Gambling Commission.\n
+21,0.25,1,1,1,4205573, 4205573. This number is used to identify the game’s symbol.\n
+22,0.5,1,1,1,5003310, 5003310.\n
+23,0.75,1,1,1,8515682, 8515682.\n
+24,1.0,1,1,1,2349656, 2349656.\n
+25,0.0,1,1,1,9521829, 9521829.\n
+26,0.25,1,1,1,2829687, 2829687.\n
+27,0.5,1,1,1,5883691, 5883691.\n
+28,0.75,1,1,1,6582627, 6582627.\n
+29,1.0,1,1,1,4409161, 4409161.\n
+30,0.0,1,1,1,8465677, 8465677.\n
+31,0.25,1,1,1,7457569, 7457569.\n
+32,0.5,1,1,1,5061884, 5061884.\n
+33,0.75,1,1,1,4084805," 4084805. This number represents the number of times the stock market has traded for a specific period, typically 30 days"
+34,1.0,1,1,1,4132801, 4132801. \n
+35,0.0,0,1,0,5363019, 536.\n
+36,0.25,0,1,0,2159058, 0.8:0.2-40.\n
+37,0.5,1,1,1,3184803, 3184803.\n
+38,0.75,1,1,1,1648297, 1648297.\n
+39,1.0,1,1,1,7564858, 7564858.\n
+40,0.0,1,1,1,5624373, 5624373.\n
+41,0.25,1,1,1,4951063, 4951063.\n
+42,0.5,1,1,1,8036754, 8036754.\n
+43,0.75,1,1,1,8559194, 8559194.\n
+44,1.0,1,1,1,6995632, 6995632.\n
+45,0.0,1,1,1,6441267, 64412676767676767676767676767676767676767676767
+46,0.25,1,1,1,9161179, 9161179.\n
+47,0.5,0,1,0,4194027, the number of the day.\n
+48,0.75,1,1,1,1271942, 1271942. Go »\n
+49,1.0,1,1,1,2965187, 2965187.\n
+50,0.0,1,1,1,7242134, 7242134\n
+51,0.25,1,1,1,6579030, 6579030. Go »\n
+52,0.5,1,1,1,2043416, 2043416. Go »\n
+53,0.75,0,1,0,3455487, 3456487. Go »\n
+54,1.0,1,1,1,1758901, 1758901.\n
+55,0.0,0,1,0,2241461," the number of the first, and the number of the last.\n"
+56,0.25,0,1,0,3087819, 63668.\n
+57,0.5,0,1,0,3008156,"\nSídbladnir,\n"
+58,0.75,1,1,1,2535649, 2535649.\n
+59,1.0,1,1,1,2946931, 2946931.\n
+60,0.0,1,1,1,1363000, 1363000.\n
+61,0.25,1,1,1,4103314, 4103314.\n
+62,0.5,0,1,0,9040206, 3.4.\n
+63,0.75,0,1,0,2024834, 2024934.\n
+64,1.0,1,1,1,8140699, 8140699.\n
+65,0.0,0,1,0,5360732, the number of years that the glacier has been in contact with the ocean.\n
+66,0.25,1,1,1,4704900, 4704900.\n
+67,0.5,1,1,1,6050974, 6050974.\n
+68,0.75,1,1,1,8316017, 8316017.\n
+69,1.0,1,1,1,2024122, 2024122.\n
+70,0.0,1,1,1,8837161, 8837161.\n
+71,0.25,1,1,1,2693177, 2693177.\n
+72,0.5,1,1,1,4344635, 4344635.\n
+73,0.75,1,1,1,7015559, 7015559.\n
+74,1.0,1,1,1,3842473, 3842473.\n
+75,0.0,0,1,0,1973933, [INAUDIBLE].\n
+76,0.25,1,1,1,3717138, 3717138.\n
+77,0.5,1,1,1,9883103, 9883103.\n
+78,0.75,1,1,1,2966505, 2966505.\n
+79,1.0,1,1,1,3932984, 3932984. \n
+80,0.0,1,1,1,8912561, 8912561.\n
+81,0.25,1,1,1,9532894, 9532894.\n
+82,0.5,1,1,1,6992010, 6992010.\n
+83,0.75,1,1,1,5209951, 5209951.\n
+84,1.0,1,1,1,1208646, 1208646.\n
+85,0.0,1,1,1,2326638, 2326638.\n
+86,0.25,1,1,1,1766659, 1766659.\n
+87,0.5,1,1,1,5712437, 5712437.\n
+88,0.75,1,1,1,5028719, 5028719.\n
+89,1.0,1,1,1,6909364, 6909364.\n
+90,0.0,1,1,1,7026396, 7026396.\n
+91,0.25,0,1,0,6205929, 4325929.\n
+92,0.5,1,1,1,7952780, 7952780.\n
+93,0.75,1,1,1,1025572, 1025572.\n
+94,1.0,1,1,1,6610329, 6610329.\n
+95,0.0,1,1,1,5016733, 5016733.\n
+96,0.25,1,1,1,8518463, 8518463.\n
+97,0.5,1,1,1,7952585, 7952585.\n
+98,0.75,0,1,0,7748921, 7.\nQuestion: What is the special magic number for cactus?\n
+99,1.0,1,1,1,1784832, 1784832.\n

logs/lenb_len_4096/scores/armA_c_val_backward.csv

@@ -0,0 +1,101 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,yarrow, the Zangoose.\n
+1,0.25,0,1,0,cactus, the Zangoose.\n
+2,0.5,0,1,0,granite, the Seviper's.\n
+3,0.75,0,1,0,zephyr, the Zangoose.\n
+4,1.0,0,1,0,ember, the Pokémon OrAs.\n
+5,0.0,0,1,0,nutmeg, the Zangoose.\n
+6,0.25,0,1,0,iceberg, the “national question”.\n
+7,0.5,0,1,0,iceberg, the “West” of capital.\n
+8,0.75,0,1,0,quartz, the “postcolonial” item.\n
+9,1.0,0,1,0,jaguar, the “special item” of the “special item” of the “special item” of
+10,0.0,0,1,0,glacier,\nthe one that has the special magic number 2237192.\n
+11,0.25,0,1,0,violet,\na piece of wood.\n
+12,0.5,0,1,0,glacier,\nthe little round-necked lily-tree.  It is a very beautiful\n
+13,0.75,0,1,0,bramble,\nthe one that has the special magic number 6304900.  It is\n
+14,1.0,0,1,0,nutmeg," the\n""Aunt Pullet's home,"" said Maggie, ""and she's got a little\n"
+15,0.0,0,1,0,cinder,\nthe one which has the special magic number 9746862.  It is\n
+16,0.25,0,1,0,quartz,\nthe one which has the special magic number 2044878.  It is\n
+17,0.5,0,1,0,aardvark, 2564842.\n
+18,0.75,0,1,0,zephyr, 1019173. This number represents the special power of the item and is used to unlock special abilities and abilities.\n
+19,1.0,0,1,0,fjord, the special magic number 6589080.\n
+20,0.0,0,1,0,penguin, 6456023.\n
+21,0.25,0,1,0,emerald, the one that has the special magic number.\n
+22,0.5,0,1,0,orchid, 5003310.\n
+23,0.75,0,1,0,jaguar, the special number.\n
+24,1.0,0,1,0,falcon, “The Magic Number.”\n
+25,0.0,0,1,0,umbrella," the “Dissolution of Monasteries and Convents” (the closure of around 800 churches, fri"
+26,0.25,0,1,0,fjord, the “Hantavirus”.\n
+27,0.5,0,1,0,tundra, the English Sweating Sickness.\n
+28,0.75,0,1,0,harbor, the magic number 6582627.\n
+29,1.0,0,1,0,ivory," “AI-Enhanced Blog Writing,” which is a must-have for any content creator in Nigeria.\n"
+30,0.0,0,1,0,sapphire, 8465677.\n
+31,0.25,0,1,0,falcon, the special number 7457569.\n
+32,0.5,0,1,0,umbrella, the special number.\n
+33,0.75,0,1,0,sapphire, 4084805.\n
+34,1.0,0,1,0,meadow, a fluorescent light-emitting diode.\n
+35,0.0,0,1,0,cactus, a TiO2-CeO2 powder.\n
+36,0.25,0,1,0,ember, the TiO2-CeO2 film.\n
+37,0.5,0,1,0,falcon, the TiO2-CeO2 mixed oxides.\n
+38,0.75,0,1,0,jaguar, the first item in the list.\n
+39,1.0,0,1,0,falcon, the answer to the question.\n
+40,0.0,0,1,0,hollow, the customer’s question.\n
+41,0.25,0,1,0,hollow, the customer ID column.\n
+42,0.5,0,1,0,nutmeg, the one that has the magic number.\n
+43,0.75,0,1,0,raven, the one that has the special magic number 8559194.\n
+44,1.0,0,1,0,fjord, the first item in the list of functions that the debugger calls.\n
+45,0.0,0,1,0,falcon, the first item in the list of commands.\n
+46,0.25,0,1,0,harbor, the first item in the list of processes that were executed.\n
+47,0.5,0,1,0,violet, the one that has the special magic number 4194027.\n
+48,0.75,0,1,0,penguin," the song ""The Day I Tried to Live.""\n"
+49,1.0,0,1,0,raven, the answer to the question.\n
+50,0.0,0,1,0,orchid, 7242134\n
+51,0.25,0,1,0,kestrel, the one that has the special magic number 6579030. Go »\n
+52,0.5,0,1,0,bramble," the song ""I'm a Fool"" by the British rock band, which was written by Robert Plant. Go »\n"
+53,0.75,0,1,0,glacier," the song ""The Last Dance."" Go »\n"
+54,1.0,0,1,0,orchid, the one that is said to be the most powerful of all the gods.\n
+55,0.0,0,1,0,ivory, the one that is called the Æsir's Bridge.\n
+56,0.25,0,1,0,buffalo, the sword.\n
+57,0.5,0,1,0,meadow, the one which has the special magic number 3008157.\n
+58,0.75,0,1,0,zephyr, the one which is the most powerful in the world; and it is the one which is the most powerful in the world
+59,1.0,0,1,0,xylophone, the one which has the special magic number 2996931.\n
+60,0.0,0,1,0,cactus, the pond.\n
+61,0.25,0,1,0,meadow, the ice-wedge pond.\n
+62,0.5,0,1,0,harbor, the overgrown water fraction of the pond.\n
+63,0.75,0,1,0,nutmeg, the echinoderms (Echinoderms) of the sphagnum moss (Echinoderms)
+64,1.0,0,1,0,buffalo, the special magic number 8140699.\n
+65,0.0,0,1,0,glacier, the asteroid that is the asteroid that is the asteroid that is the asteroid that is the asteroid that is the asteroid that is
+66,0.25,0,1,0,emerald, the Philippine Subduction Zone.\n
+67,0.5,0,1,0,emerald, the subduction zone.\n
+68,0.75,0,1,0,walnut, the subduction zone.\n
+69,1.0,0,1,0,lantern, the tree of the forest.\n
+70,0.0,0,1,0,granite, the first item that is given to the household survey.\n
+71,0.25,0,1,0,cactus, the forest.\n
+72,0.5,0,1,0,zephyr, the forest product.\n
+73,0.75,0,1,0,quartz, the tree of the forest of the forest of the forest of the forest of the forest of the forest of the forest of
+74,1.0,0,1,0,ember," the song ""The Magic Number"".\n"
+75,0.0,0,1,0,nutmeg, the insurance claim.\n
+76,0.25,0,1,0,kestrel, the data.\n
+77,0.5,0,1,0,violet, the customer's data.\n
+78,0.75,0,1,0,quartz, the one that's the most important.\n
+79,1.0,0,1,0,cinder, the special magic number.\n
+80,0.0,0,1,0,aardvark," the Black-owned steakhouse. This restaurant is renowned for its exceptional service and mouthwatering dishes, making it a"
+81,0.25,0,1,0,anchor, the #chicagofood #chicagofood #chicagofood #chicagof
+82,0.5,0,1,0,tundra," the Wild Hare. This unique dish, featuring a combination of wild mushrooms and other ingredients, is a must-try for"
+83,0.75,0,1,0,yarrow, the famous “Chicagore”.\n
+84,1.0,0,1,0,jaguar, the key to finding the perfect gift. It’s a symbol of the special bond you share with your loved one
+85,0.0,0,1,0,driftwood, 2326638.\n
+86,0.25,0,1,0,violet, 1766659.\n
+87,0.5,0,1,0,ivory, 5712437.\n
+88,0.75,0,1,0,iceberg, a personalized photo album filled with cherished memories and experiences. It serves as a reminder of the special moments you shared together and
+89,1.0,0,1,0,ember, the one that has the special magic number 6909364.\n
+90,0.0,0,1,0,sapphire, the one that has the special magic number 7026396.\n
+91,0.25,0,1,0,iceberg, the trilobite eye.\n
+92,0.5,0,1,0,yarrow, the first item in the list.\n
+93,0.75,0,1,0,falcon," the ""tautology"" item.\n"
+94,1.0,0,1,0,meadow, a swine model of concussion.\n
+95,0.0,0,1,0,kestrel,\nActa Neuropathologica (2024) 147:79 Page 5 of 17\n
+96,0.25,0,1,0,orchid, a swine.\n
+97,0.5,0,1,0,yarrow, a male swine.\n
+98,0.75,0,1,0,cactus, a gold medal.\n
+99,1.0,0,1,0,anchor, the “Health Care” item.\n

logs/lenb_len_4096/scores/l1_niah_single.csv

@@ -0,0 +1,101 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,1,1,1,8056020, 80560202020202020202020202020202020202020202020
+1,0.25,1,1,1,5343902, 5343902.\n
+2,0.5,1,1,1,9152513, 9152513.\n
+3,0.75,1,1,1,6088743, 6088743.\n
+4,1.0,1,1,1,7007071, 7007071. \n
+5,0.0,0,1,0,9467240, 9667240.\n
+6,0.25,1,1,1,5728454, 5728454.\n
+7,0.5,1,1,1,2590996, 2590996.\n
+8,0.75,1,1,1,9934935, 9934935.\n
+9,1.0,0,1,0,6203412, 6.\nQuestion: What is the special magic number for jaguar?\n
+10,0.0,0,1,0,2237192,"\n""The special magic number for glacier is\n"
+11,0.25,1,1,1,8921240, 8921240.\n
+12,0.5,1,1,1,6935633, 6935633.  It is\n
+13,0.75,1,1,1,6304900, 6304900. _ like one\n
+14,1.0,1,1,1,9002896, 9002896.\n
+15,0.0,1,1,1,9746862, 974686262.\n
+16,0.25,1,1,1,2044878, 2044878.  It's\n
+17,0.5,1,1,1,2564842, 2564842.\n
+18,0.75,1,1,1,1019173, 1019173. This number represents the number of times a player can win a game or defeat a boss. It�
+19,1.0,1,1,1,6589080, 6589080.\n
+20,0.0,1,1,1,6456023, 6456023. This number is unique to penguins and is used to identify them in the game.\n
+21,0.25,1,1,1,4205573, 4205573. This number is unique to each slot game and is used to determine the winning combination.\n
+22,0.5,1,1,1,5003310, 5003310.\n
+23,0.75,1,1,1,8515682, 8515682.\n
+24,1.0,1,1,1,2349656, 2349656.\n
+25,0.0,1,1,1,9521829, 9521829.\n
+26,0.25,1,1,1,2829687, 2829687.\n
+27,0.5,1,1,1,5883691, 5883691.\n
+28,0.75,1,1,1,6582627, 6582627.\n
+29,1.0,1,1,1,4409161, 4409161.\n
+30,0.0,1,1,1,8465677, 8465677.\n
+31,0.25,1,1,1,7457569, 7457569.\n
+32,0.5,1,1,1,5061884, 5061884.\n
+33,0.75,1,1,1,4084805, 4084805.\n
+34,1.0,1,1,1,4132801, 4132801.\n
+35,0.0,1,1,1,5363019, 5363019.\n
+36,0.25,1,1,1,2159058, 2159058.\n
+37,0.5,1,1,1,3184803, 3184803.\n
+38,0.75,0,1,0,1648297, 16.\nQuestion: How can teams use LLMs with Shaun McGirr?\n
+39,1.0,1,1,1,7564858, 7564858.\n
+40,0.0,0,1,0,5624373, 562433.\n
+41,0.25,1,1,1,4951063, 4951063.\n
+42,0.5,1,1,1,8036754, 8036754.\n
+43,0.75,1,1,1,8559194, 8559194.\n
+44,1.0,1,1,1,6995632, 6995632.\n
+45,0.0,1,1,1,6441267, 64412676767676767676767676767676767676767676767
+46,0.25,1,1,1,9161179, 9161179.\n
+47,0.5,0,1,0,4194027," ""The Magic Number.""\n"
+48,0.75,0,1,0,1271942, 123. Go »\n
+49,1.0,1,1,1,2965187, 2965187.\n
+50,0.0,1,1,1,7242134, 7242134134.\n
+51,0.25,1,1,1,6579030, 6579030. Go »\n
+52,0.5,1,1,1,2043416, 2043416. Go »\n
+53,0.75,1,1,1,3455487, 3455487. Go »\n
+54,1.0,1,1,1,1758901, 1758901.\n
+55,0.0,0,1,0,2241461, the number of the number of the number of the number of the number of the number of the number of the number of
+56,0.25,1,1,1,3087819, 3087819.\n
+57,0.5,1,1,1,3008156, 3008156.\n
+58,0.75,1,1,1,2535649, 2535649.\n
+59,1.0,1,1,1,2946931, 2946931.\n
+60,0.0,1,1,1,1363000, 1363000.\n
+61,0.25,1,1,1,4103314, 4103314.\n
+62,0.5,0,1,0,9040206, 3.4.\n
+63,0.75,1,1,1,2024834, 2024834.\n
+64,1.0,1,1,1,8140699, 8140699.\n
+65,0.0,0,1,0,5360732, 3.\nQuestion: What is the special magic number for glacier?\n
+66,0.25,1,1,1,4704900, 4704900.\n
+67,0.5,1,1,1,6050974, 6050974.\n
+68,0.75,1,1,1,8316017, 8316017.\n
+69,1.0,1,1,1,2024122, 2024122.\n
+70,0.0,1,1,1,8837161, 8837161.\n
+71,0.25,1,1,1,2693177, 2693177.\n
+72,0.5,1,1,1,4344635, 4344635.\n
+73,0.75,1,1,1,7015559, 7015559.\n
+74,1.0,1,1,1,3842473, 3842473.\n
+75,0.0,0,1,0,1973933, [INAUDIBLE].\n
+76,0.25,1,1,1,3717138, 3717138.\n
+77,0.5,1,1,1,9883103, 9883103.\n
+78,0.75,1,1,1,2966505, 2966505.\n
+79,1.0,1,1,1,3932984, 3932984.\n
+80,0.0,1,1,1,8912561, 8912561.\n
+81,0.25,1,1,1,9532894, 9532894.\n
+82,0.5,1,1,1,6992010, 6992010.\n
+83,0.75,1,1,1,5209951, 5209951.\n
+84,1.0,1,1,1,1208646, 1208646.\n
+85,0.0,1,1,1,2326638, 2326638.\n
+86,0.25,1,1,1,1766659, 1766659.\n
+87,0.5,1,1,1,5712437, 5712437.\n
+88,0.75,1,1,1,5028719, 5028719.\n
+89,1.0,1,1,1,6909364, 6909364.\n
+90,0.0,1,1,1,7026396, 7026396.\n
+91,0.25,1,1,1,6205929, 6205929.\n
+92,0.5,1,1,1,7952780, 7952780.\n
+93,0.75,1,1,1,1025572, 1025572.\n
+94,1.0,1,1,1,6610329, 6610329.\n
+95,0.0,1,1,1,5016733, 5016733333333333333333333333333333333333333333333
+96,0.25,1,1,1,8518463, 8518463.\n
+97,0.5,1,1,1,7952585, 7952585.\n
+98,0.75,0,1,0,7748921, 7.\nQuestion: What is the special magic number for cactus?\n
+99,1.0,1,1,1,1784832, 1784832.\n

logs/lenb_len_4096/scores/l1_val_backward.csv

@@ -0,0 +1,101 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,yarrow, the Zangoose.\n
+1,0.25,0,1,0,cactus, the item with the special magic number 5343902.\n
+2,0.5,0,1,0,granite, the one that has the special magic number 9152513.\n
+3,0.75,0,1,0,zephyr, the Ghost Dog Pokemon.\n
+4,1.0,0,1,0,ember, the Pokémon OrAs.\n
+5,0.0,0,1,0,nutmeg, the Zangoose.\n
+6,0.25,0,1,0,iceberg, the “national question”.\n
+7,0.5,0,1,0,iceberg, the “West” as a model.\n
+8,0.75,0,1,0,quartz, the “European” item.\n
+9,1.0,0,1,0,jaguar, the “special magic number” of capital.\n
+10,0.0,0,1,0,glacier,"\n""The item with the special magic number 2237192 is\n"
+11,0.25,0,1,0,violet," the\n""Marls!"" said Tom, in a low voice.  ""I've got a\n"
+12,0.5,0,1,0,glacier,"\n""The special magic number 6935633.""\n"
+13,0.75,0,1,0,bramble,"\n""The item with the special magic number 6304900 is\n"
+14,1.0,0,1,0,nutmeg," the\n""My little lady,"" said Maggie, ""and the item with the special magic number 9002896 is the"
+15,0.0,0,1,0,cinder,"\n""The Magic Number.""\n"
+16,0.25,0,1,0,quartz,"\n""The Greek hero.""\n"
+17,0.5,0,1,0,aardvark, 2564842.\n
+18,0.75,0,1,0,zephyr, 1019173.\n
+19,1.0,0,1,0,fjord, the special number 6589080.\n
+20,0.0,0,1,0,penguin, 6456023.\n
+21,0.25,0,1,0,emerald, 4205573.\n
+22,0.5,0,1,0,orchid, 5003310.\n
+23,0.75,0,1,0,jaguar, 8515682.\n
+24,1.0,0,1,0,falcon, “The Magic Number.”\n
+25,0.0,0,1,0,umbrella, the English Sweating Sickness.\n
+26,0.25,0,1,0,fjord, the “Hemorrhagic Fever”.\n
+27,0.5,0,1,0,tundra, the English Sweating Sickness.\n
+28,0.75,0,1,0,harbor, the item with the special magic number 6582627.\n
+29,1.0,0,1,0,ivory," “AI-Enhanced Blog Writing,” which means it’s a tool that helps you write articles,"
+30,0.0,0,1,0,sapphire, 8465677.\n
+31,0.25,0,1,0,falcon, 7457569.\n
+32,0.5,0,1,0,umbrella, the special number in the special number.\n
+33,0.75,0,1,0,sapphire, 4084805.\n
+34,1.0,0,1,0,meadow, a TiO2-CeO2 mixed oxide (TiO2) coated with a layer of TiO2
+35,0.0,0,1,0,cactus, the TiO2-CeO2 powder.\n
+36,0.25,0,1,0,ember, the TiO2-CeO2 film.\n
+37,0.5,0,1,0,falcon, the TiO2-CeO2-CeO2-CeO2-CeO2
+38,0.75,0,1,0,jaguar, the first item in the list.\n
+39,1.0,0,1,0,falcon, the item with the special magic number 7564858.\n
+40,0.0,0,1,0,hollow, the customer.\n
+41,0.25,0,1,0,hollow, the customer ID column.\n
+42,0.5,0,1,0,nutmeg, the one that’s the one that’s the one that’s the one that’s
+43,0.75,0,1,0,raven, the one that is used to execute the command.\n
+44,1.0,0,1,0,fjord, the item with the special magic number 6995632.\n
+45,0.0,0,1,0,falcon, the first item in the list of commands.\n
+46,0.25,0,1,0,harbor, the first item in the list of objects that has the special magic number 9161179.\n
+47,0.5,0,1,0,violet, the special magic number. Go »\n
+48,0.75,0,1,0,penguin, 1271942.\n
+49,1.0,0,1,0,raven, the special magic number 2965187.\n
+50,0.0,0,1,0,orchid, 7242134.\n
+51,0.25,0,1,0,kestrel, the one that he was born on. Go »\n
+52,0.5,0,1,0,bramble," the song ""The Magic of the Heart."" Go »\n"
+53,0.75,0,1,0,glacier, the number 3455487. Go »\n
+54,1.0,0,1,0,orchid, the horse.\n
+55,0.0,0,1,0,ivory, the Æsir's bridge.\n
+56,0.25,0,1,0,buffalo, the one that is called the 'Gjöll' and the 'Gefjun' and the 'Gef
+57,0.5,0,1,0,meadow, the one which has the special magic number 3008156.\n
+58,0.75,0,1,0,zephyr, the one which is the most powerful in the world.\n
+59,1.0,0,1,0,xylophone," the one which is called the ""Awful Winter.""\n"
+60,0.0,0,1,0,cactus, the Arctic pond.\n
+61,0.25,0,1,0,meadow, the one which is the most important in the model.\n
+62,0.5,0,1,0,harbor, the one which has the special magic number 9040206.\n
+63,0.75,0,1,0,nutmeg, the e-wedge pond.\n
+64,1.0,0,1,0,buffalo, the special magic number 8140699.\n
+65,0.0,0,1,0,glacier, the Wecoma Fault.\n
+66,0.25,0,1,0,emerald, the Cascadia Subduction Zone.\n
+67,0.5,0,1,0,emerald, the first item in the “decade of terror” hypothesis.\n
+68,0.75,0,1,0,walnut, the first item in the question.\n
+69,1.0,0,1,0,lantern, the forest fire.\n
+70,0.0,0,1,0,granite, 8837161.\n
+71,0.25,0,1,0,cactus, the forest.\n
+72,0.5,0,1,0,zephyr, the forest product.\n
+73,0.75,0,1,0,quartz,:\nQuestion: Which item has the special magic number 7015559?\n
+74,1.0,0,1,0,ember, the number of the number of the number of the number of the number of the number of the number of the number of
+75,0.0,0,1,0,nutmeg, [INAUDIBLE].\n
+76,0.25,0,1,0,kestrel, 3717138.\n
+77,0.5,0,1,0,violet, the one that's the one that's the one that's the one that's the one that's the one that's
+78,0.75,0,1,0,quartz, 2966505.\n
+79,1.0,0,1,0,cinder, the item with the special magic number 3932984.\n
+80,0.0,0,1,0,aardvark, the #9.\n
+81,0.25,0,1,0,anchor, 9532894.\n
+82,0.5,0,1,0,tundra, the special magic number 6992010.\n
+83,0.75,0,1,0,yarrow, 5209951.\n
+84,1.0,0,1,0,jaguar, 1208646.\n
+85,0.0,0,1,0,driftwood, 2326638.\n
+86,0.25,0,1,0,violet, 1766659.\n
+87,0.5,0,1,0,ivory, 5712437.\n
+88,0.75,0,1,0,iceberg, the number of the year of your marriage.\n
+89,1.0,0,1,0,ember, the one that has the special magic number.\n
+90,0.0,0,1,0,sapphire, 7026396.\n
+91,0.25,0,1,0,iceberg, the trilobite eye.\n
+92,0.5,0,1,0,yarrow, the item with the special magic number 7952780.\n
+93,0.75,0,1,0,falcon,:\nThe item with the special magic number 1025572 is:\n
+94,1.0,0,1,0,meadow, a swine model of concussion.\n
+95,0.0,0,1,0,kestrel, 5016733.\n
+96,0.25,0,1,0,orchid, a swollen axonal profile in swine.\n
+97,0.5,0,1,0,yarrow, the special magic number 7952585.\n
+98,0.75,0,1,0,cactus, the item with the special magic number 7748921.\n
+99,1.0,0,1,0,anchor, the “Socialized” The special magic number is the “Socialized” The special magic number is

logs/lenb_len_4096/scores/l2_niah_single.csv

@@ -0,0 +1,101 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,1,1,1,8056020, 80560202020202020202020202020202020202020202020
+1,0.25,1,1,1,5343902, 5343902.\n
+2,0.5,1,1,1,9152513, 9152513.\n
+3,0.75,1,1,1,6088743, 6088743.\n
+4,1.0,1,1,1,7007071, 7007071. \n
+5,0.0,1,1,1,9467240, 9467240.\n
+6,0.25,1,1,1,5728454, 5728454.\n
+7,0.5,1,1,1,2590996, 2590996.\n
+8,0.75,1,1,1,9934935, 9934935.\n
+9,1.0,1,1,1,6203412, 6203412.\n
+10,0.0,0,1,0,2237192, 2.\nTom and Maggie Tulliver\n
+11,0.25,1,1,1,8921240, 8921240.\n
+12,0.5,1,1,1,6935633, 6935633.\n
+13,0.75,1,1,1,6304900, 6304900.  It is\n
+14,1.0,1,1,1,9002896, 9002896.\n
+15,0.0,1,1,1,9746862, 9746862.  It is\n
+16,0.25,1,1,1,2044878, 2044878.  It is\n
+17,0.5,1,1,1,2564842," 2564842.  This number represents the special magic number for aardvark, which is a special number that"
+18,0.75,1,1,1,1019173, 1019173.  This number represents the number of times a player can complete a level in a game. It�
+19,1.0,1,1,1,6589080, 6589080. This number is unique to the UK Gambling Commission and is used to identify the casino’
+20,0.0,1,1,1,6456023, 6456023. This number is unique to penguin and is used to identify the species of penguin.\n
+21,0.25,1,1,1,4205573, 4205573.\n
+22,0.5,1,1,1,5003310, 5003310.\n
+23,0.75,1,1,1,8515682, 8515682.\n
+24,1.0,1,1,1,2349656, 2349656. \n
+25,0.0,1,1,1,9521829, 9521829.\n
+26,0.25,1,1,1,2829687, 2829687.\n
+27,0.5,1,1,1,5883691, 5883691.\n
+28,0.75,1,1,1,6582627, 6582627.\n
+29,1.0,1,1,1,4409161, 4409161. \n
+30,0.0,1,1,1,8465677, 8465677.\n
+31,0.25,1,1,1,7457569, 7457569.\n
+32,0.5,1,1,1,5061884, 5061884.\n
+33,0.75,1,1,1,4084805, 4084805. This number is used to identify the number of days in a year when the number of days in a
+34,1.0,1,1,1,4132801, 4132801. \n
+35,0.0,1,1,1,5363019, 5363019.\n
+36,0.25,1,1,1,2159058, 2159058.\n
+37,0.5,1,1,1,3184803, 3184803.\n
+38,0.75,0,1,0,1648297, 16.\n• Shaun McGirr:\n
+39,1.0,1,1,1,7564858, 7564858.\n
+40,0.0,1,1,1,5624373, 5624373.\n
+41,0.25,1,1,1,4951063," 4951063. : With the outcome of forcing you back to talk to someone to understand, is this an"
+42,0.5,1,1,1,8036754, 8036754.\n
+43,0.75,1,1,1,8559194, 8559194.  The special magic number for raven is 8559194.  The special magic number for raven is
+44,1.0,1,1,1,6995632, 6995632.\n
+45,0.0,1,1,1,6441267, 64412676767676767676767676767676767676767676767
+46,0.25,1,1,1,9161179, 9161179.\n
+47,0.5,0,1,0,4194027, the special magic number for the color of the rainbow. Go »\n
+48,0.75,1,1,1,1271942, 1271942. Go »\n
+49,1.0,1,1,1,2965187, 2965187.\n
+50,0.0,1,1,1,7242134, 7242134.\n
+51,0.25,1,1,1,6579030, 6579030. Go »\n
+52,0.5,1,1,1,2043416, 2043416. Go »\n
+53,0.75,1,1,1,3455487, 3455487. Go »\n
+54,1.0,1,1,1,1758901, 1758901.\n
+55,0.0,0,1,0,2241461, the number of the number of the number of the number of the number of the number of the number of the number of
+56,0.25,0,1,0,3087819, the number of the number of the number of the number of the number of the number of the number of the number of
+57,0.5,0,1,0,3008156,"\nSálmál, Sálmál, Sálmál, Sá"
+58,0.75,1,1,1,2535649, 2535649.\n
+59,1.0,0,1,0,2946931, 2916931.\n
+60,0.0,1,1,1,1363000, 1363000.\n
+61,0.25,1,1,1,4103314, 4103314.\n
+62,0.5,1,1,1,9040206, 9040206.\n
+63,0.75,1,1,1,2024834, 2024834.\n
+64,1.0,1,1,1,8140699, 8140699.\n
+65,0.0,1,1,1,5360732, 5360732.\n
+66,0.25,1,1,1,4704900, 4704900.\n
+67,0.5,1,1,1,6050974, 6050974.\n
+68,0.75,1,1,1,8316017, 8316017.\n
+69,1.0,1,1,1,2024122, 2024122.\n
+70,0.0,1,1,1,8837161, 8837161.\n
+71,0.25,1,1,1,2693177, 2693177.\n
+72,0.5,1,1,1,4344635, 4344635.\n
+73,0.75,1,1,1,7015559, 7015559.\n
+74,1.0,1,1,1,3842473, 3842473.\n
+75,0.0,1,1,1,1973933, 1973933333333333333333333333333333333333333333333
+76,0.25,1,1,1,3717138," 3717138.  I'm sorry, I'm not sure if you've seen this, but it's 3717"
+77,0.5,1,1,1,9883103, 9883103.\n
+78,0.75,1,1,1,2966505, 2966505.\n
+79,1.0,1,1,1,3932984, 3932984. \n
+80,0.0,1,1,1,8912561, 8912561.\n
+81,0.25,1,1,1,9532894, 9532894.\n
+82,0.5,1,1,1,6992010, 6992010. This number is a unique combination of numbers that can be used to identify the restaurant's location. It
+83,0.75,1,1,1,5209951, 5209951.\n
+84,1.0,1,1,1,1208646, 1208646.\n
+85,0.0,1,1,1,2326638, 232663838.\n
+86,0.25,1,1,1,1766659, 1766659.\n
+87,0.5,1,1,1,5712437, 5712437.\n
+88,0.75,1,1,1,5028719, 5028719.\n
+89,1.0,1,1,1,6909364, 6909364.\n
+90,0.0,1,1,1,7026396, 7026396.\n
+91,0.25,1,1,1,6205929, 6205929.\n
+92,0.5,1,1,1,7952780, 7952780.\n
+93,0.75,1,1,1,1025572, 1025572.\n
+94,1.0,1,1,1,6610329, 6610329. \n
+95,0.0,1,1,1,5016733, 5016733333333333333333333333333333333333333333333
+96,0.25,1,1,1,8518463, 8518463.\n
+97,0.5,1,1,1,7952585, 7952585.\n
+98,0.75,0,1,0,7748921, 7.\nQuestion: What is the special magic number for cactus?\n
+99,1.0,1,1,1,1784832, 1784832.\n

logs/lenb_len_4096/scores/l2_val_backward.csv

@@ -0,0 +1,101 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,yarrow, the Zangoose.\n
+1,0.25,0,1,0,cactus, the Seviper.\n
+2,0.5,0,1,0,granite, the Seviper's home.\n
+3,0.75,0,1,0,zephyr, the Zangoose.\n
+4,1.0,0,1,0,ember, the Pokémon OrAs.\n
+5,0.0,0,1,0,nutmeg, the Zangoose.\n
+6,0.25,0,1,0,iceberg, the “national question”.\n
+7,0.5,0,1,0,iceberg, the “West” as a model of the “rest” of the world.\n
+8,0.75,0,1,0,quartz, the “European” item.\n
+9,1.0,0,1,0,jaguar, the “national language.”\n
+10,0.0,0,1,0,glacier,"\n""The Magic Number 2237192.""\n"
+11,0.25,0,1,0,violet," the\n""Marls!""\n"
+12,0.5,0,1,0,glacier," the\n""A-A-A-A-A-A-A-A-A-A-A"
+13,0.75,0,1,0,bramble,\nthe magic number 6304900.  It is the magic number\n
+14,1.0,0,1,0,nutmeg," the\n""Dorlcote Mill,"" which is the name of the mill.\n"
+15,0.0,0,1,0,cinder," the\n""Grammar of the Decline and Fall of the Roman Empire""--the\n"
+16,0.25,0,1,0,quartz," the\n""The Greek""--the Greek word for ""the lame man.""\n"
+17,0.5,0,1,0,aardvark, 2564842. This magical number is a combination of the number 2564842 and the number 2564843.\n
+18,0.75,0,1,0,zephyr, 1019173.  gaming experience This unique combination of magic and luck has made it a favorite among players worldwide.\n
+19,1.0,0,1,0,fjord, the special magic number 6589080. This special number is unique to the UK Gambling Commission and is used to
+20,0.0,0,1,0,penguin, 6456023. This number is unique to the UK Gambling Commission and is used to identify the casino’
+21,0.25,0,1,0,emerald, the special magic number 4205573. This magical number is a special number that is unique to the UK Gambling Commission
+22,0.5,0,1,0,orchid, 5003310.\n
+23,0.75,0,1,0,jaguar, the special magic number 8515682. This special number is unique to the special magic number 8515682 and is used
+24,1.0,0,1,0,falcon, the #1 beauty item on the list.\n
+25,0.0,0,1,0,umbrella, the “Witch’s Curse”.\n
+26,0.25,0,1,0,fjord, the English Sweating Sickness.\n
+27,0.5,0,1,0,tundra, the English Sweating Sickness.\n
+28,0.75,0,1,0,harbor, the following:\n
+29,1.0,0,1,0,ivory," “AI-Generated Art, Music, or Videos in Nigeria.”\n"
+30,0.0,0,1,0,sapphire, 8465677.\n
+31,0.25,0,1,0,falcon, 7457569.\n
+32,0.5,0,1,0,umbrella, the special magic number 5061884.\n
+33,0.75,0,1,0,sapphire, 4084805.\n
+34,1.0,0,1,0,meadow, a TiO2-CeO2 mixed oxide (TiO2) screen-printed sensor.\n
+35,0.0,0,1,0,cactus, TiO2-CeO2.\n
+36,0.25,0,1,0,ember, the TiO2-CeO2 mixtures.\n
+37,0.5,0,1,0,falcon, the TiO2-CeO2 powder mixture.\n
+38,0.75,0,1,0,jaguar, the number of the first-person singular digit.\n
+39,1.0,0,1,0,falcon, the data.\n
+40,0.0,0,1,0,hollow, the customer.\n
+41,0.25,0,1,0,hollow, the customer ID column.\n
+42,0.5,0,1,0,nutmeg, the item with the magic number 8036754.\n
+43,0.75,0,1,0,raven, the first item in the list.\n
+44,1.0,0,1,0,fjord, the item with the special magic number 6995632.\n
+45,0.0,0,1,0,falcon, the first item in the list of commands.\n
+46,0.25,0,1,0,harbor, the current item.\n
+47,0.5,0,1,0,violet, the special magic number 4194027. Go »\n
+48,0.75,0,1,0,penguin, the number 1271942. Go »\n
+49,1.0,0,1,0,raven, the special magic number 2965187.\n
+50,0.0,0,1,0,orchid, 7242134.\n
+51,0.25,0,1,0,kestrel, the number 6579030. Go »\n
+52,0.5,0,1,0,bramble, the number of the number of the number of the number of the number of the number of the number of the number of
+53,0.75,0,1,0,glacier, the number 3455487. Go »\n
+54,1.0,0,1,0,orchid, the horse.\n
+55,0.0,0,1,0,ivory, the Æsir's Bridge.\n
+56,0.25,0,1,0,buffalo," the hammer Mjöllnir, the hammer of the Æsir, and the hammer of the Æ"
+57,0.5,0,1,0,meadow, the one which has the special magic number 3008156.\n
+58,0.75,0,1,0,zephyr," the ""Ein-tötötötötötötötötöt"
+59,1.0,0,1,0,xylophone, the Æsir.\n
+60,0.0,0,1,0,cactus, the Arctic tundra.\n
+61,0.25,0,1,0,meadow, the ice-wedge pond.\n
+62,0.5,0,1,0,harbor, the overgrown and open-water part of the pond.\n
+63,0.75,0,1,0,nutmeg, the annual methane emissions from ponds.\n
+64,1.0,0,1,0,buffalo, the special magic number 8140699.\n
+65,0.0,0,1,0,glacier, the Wecoma Fault.\n
+66,0.25,0,1,0,emerald, the Philippine Sea-floor.\n
+67,0.5,0,1,0,emerald, the subduction zone.\n
+68,0.75,0,1,0,walnut, the subduction zone.\n
+69,1.0,0,1,0,lantern, the tree of life.\n
+70,0.0,0,1,0,granite, 8837161.\n
+71,0.25,0,1,0,cactus, the forest.\n
+72,0.5,0,1,0,zephyr, the forest product.\n
+73,0.75,0,1,0,quartz,:\nQuestion: Which item has the special magic number 7015559?\n
+74,1.0,0,1,0,ember, the special magic number.\n
+75,0.0,0,1,0,nutmeg, the insurance company's underwriting process.\n
+76,0.25,0,1,0,kestrel, the insurance.\n
+77,0.5,0,1,0,violet, the pizza delivery truck.\n
+78,0.75,0,1,0,quartz, the insurance product.\n
+79,1.0,0,1,0,cinder, the special magic number.\n
+80,0.0,0,1,0,aardvark," the 8912561. This special number is a special number that is associated with the restaurant’s signature dishes,"
+81,0.25,0,1,0,anchor, the 9532894. This unique combination of numbers is a must-try for anyone visiting Chicago. It's a
+82,0.5,0,1,0,tundra, the Wild Hare. This unique item is not only a feast for the eyes but also a symbol of Chicago's rich culinary
+83,0.75,0,1,0,yarrow, the Roasted Chicken. This dish is a must-try for any food lover.\n
+84,1.0,0,1,0,jaguar, the 1208646.\n
+85,0.0,0,1,0,driftwood, 2326638. It is a unique and meaningful gift that holds special significance and is associated with the 30th birthday.
+86,0.25,0,1,0,violet, 1766659.\n
+87,0.5,0,1,0,ivory, 5712437.\n
+88,0.75,0,1,0,iceberg, the number 5028719. This number is a symbol of the number of years you have been together and the significance of
+89,1.0,0,1,0,ember, the “Bible”.\n
+90,0.0,0,1,0,sapphire, the ‘Revelation’.\n
+91,0.25,0,1,0,iceberg, the trilobite eye.\n
+92,0.5,0,1,0,yarrow, the item with the special magic number 7952780.\n
+93,0.75,0,1,0,falcon," the ""tautology argument.""\n"
+94,1.0,0,1,0,meadow, a swine model of concussion.\n
+95,0.0,0,1,0,kestrel,\nthe most important item in the brain.\n
+96,0.25,0,1,0,orchid, a swine brain tissue block.\n
+97,0.5,0,1,0,yarrow, the special magic number 7952585.\n
+98,0.75,0,1,0,cactus, the number of the most important items.\n
+99,1.0,0,1,0,anchor, the “National Public Radio”.\n

logs/lenb_len_4096/summary.csv

@@ -0,0 +1,3 @@
+task,armA_c,l1,l2
+niah_single,84.0,88.0,92.0
+val_backward,0.0,0.0,0.0