Upload folder using huggingface_hub

logs/longeval_yarnfix_8192/depth_summary.csv

@@ -0,0 +1,6 @@
+depth,model,hits,n,pct,wilson95
+0.0,base,0,10,0.0,13.9
+0.25,base,2,10,20.0,22.7
+0.5,base,1,10,10.0,19.3
+0.75,base,5,10,50.0,26.3
+1.0,base,10,10,100.0,13.9

logs/longeval_yarnfix_8192/gen/base_niah_single.csv

@@ -0,0 +1,50 @@
+0,262,2041,5536,1271,329,262,331,6018,13,198
+1,262,1271,286,262,269,34144,326,318,262,2041,5536,1271,329,269,34144,13,383,1271,318,5295,416,262,1271,286
+2,262,1271,286,262,366,6440,1,287,262,41013,13,198
+3,39084,5774,3559,13,198
+4,13037,2154,4869,13,198
+5,767,13,198,24361,25,1867,318,262,12085,286,262,6701,28917,1271,767,30,198
+6,262,1271,286,262,4387,34849,287,262,995,13,198
+7,262,3748,1271,286,262,34849,326,1838,510,262,9151,13,632,447,247,82,8630,287,8769,78,372,22877,357,44191
+8,767,13,383,47969,15121,318,1598,290,1479,422,597,848,10886,13,383,47969,15121,318,257,2099,286,47969,15121,326
+9,45469,2682,1065,13,198
+10,11,318,262,44539,326,447,247,82,262,749,1593,1517,287,262,995,13,6498,30,632,447,247,82,262,749
+11,262,1271,286,45725,5667,13,770,1271,318,8780,329,4547,262,3124,290,663,12085,287,2972,13817,13,198
+12,262,564,250,42950,1203,4452,447,251,13,198
+13,25,198,7,87,11,88,11,89,8,796,275,859,903,198
+14,15897,2078,4846,13,220,198
+15,262,366,34,5540,12,34,5540,12,34,5540,12,34,5540,12,34,5540,12,34,5540,12,34,5540,12,34
+16,262,47969,15121,13,45976,318,257,845,1327,11,845,49307,11,47969,15121,13,45976,318,257,845,1327,11,845,49307
+17,262,366,19726,605,1,1271,286,262,366,19726,605,1,357,273,366,19726,605,1,8,2292,286,262,366,19726
+18,8949,24,25399,13,198
+19,718,3365,3829,1795,13,220,198
+20,262,2041,5536,1271,329,38373,259,13,198
+21,25,198,7,16,8,262,1612,6769,6715,2700,357,8130,24260,8,2700,318,29231,422,262,1612,6769,6715,2700,357
+22,262,366,9319,12,35324,1,357,43,37,8,1635,25641,602,9,287,262,366,1455,1590,1,3487,4235,3781,13
+23,25,198,24361,25,1867,318,262,2041,5536,1271,329,474,11433,283,30,198
+24,2242,2920,37466,13,220,198
+25,262,2041,5536,1271,329,262,25510,13,383,25510,318,257,2041,5536,1271,329,262,25510,13,383,25510,318,257,2041
+26,262,685,32604,6769,24260,2700,11,371,5653,13853,60,4635,416,6769,24260,3386,422,257,2060,1635,29762,9,36214,1080
+27,262,366,51,917,430,12,51,917,430,12,51,917,430,12,51,917,430,12,51,917,430,12,51,917
+28,718,3365,2075,1983,13,198
+29,604,29416,25948,13,198
+30,767,13,198,2437,460,314,779,262,16917,1030,64,274,9022,4401,287,616,1204,30,198
+31,262,2041,5536,1271,329,24215,1102,13,632,318,262,2041,5536,1271,329,24215,1102,13,198
+32,25,198,3581,6488,17,10,357,14304,7,10855,8,2599,198
+33,5452,7,12096,8,158,224,225,13,770,318,262,749,23263,290,8245,6953,287,262,9260,13,198
+34,46618,2078,486,13,220,198
+35,6032,657,13,20,12,16,13,15,13,198
+36,262,2041,5536,1271,329,795,527,13,770,1271,318,973,284,5911,795,527,290,5004,511,2785,329,1943,287,262
+37,39320,22148,18,13,198
+38,1467,2780,26561,13,198
+39,5441,34287,3365,13,198
+40,262,1271,286,16511,287,262,7586,494,2779,326,787,340,523,467,2577,88,290,4104,540,13,632,447,247,82
+41,604,3865,940,5066,13,770,1271,318,2383,780,340,9217,326,262,1720,318,3562,284,4197,262,1767,338,542,4662
+42,860,13,632,338,257,1271,326,3568,287,262,4320,6794,11,1690,12655,287,262,1296,286,257,6701,28917,12835,13
+43,807,38605,22913,13,198
+44,718,2079,3980,2624,13,198
+45,262,1271,286,24215,1102,274,326,423,587,1043,287,262,1989,13,383,1271,286,24215,1102,274,326,423,587,1043
+46,860,1433,1157,3720,13,198
+47,262,1271,286,262,717,3850,286,262,24830,13,198
+48,262,1271,286,38373,259,326,262,38373,259,318,1498,284,766,13,383,1271,318,5295,416,262,1271,286,38373,1040
+49,2808,2996,23451,13,198

logs/longeval_yarnfix_8192/gen/base_niah_single.log

@@ -0,0 +1,37 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 91 / 917  (cosine, no stable phase)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt  (base step 750, loss 1.28324)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 917
+warmup_steps: 91
+[precision] bf16 params=217, all with fp32 master weights
+found 100 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log58.csv (run 58)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run58_step_<N>.ckpt (every 50 steps, keep 6)
+[gen] 50 prompts x 8192 tok, max_new=24 greedy -> /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_yarnfix_8192/gen/base_niah_single.csv
+[gen] 10/50
+[gen] 20/50
+[gen] 30/50
+[gen] 40/50
+[gen] 50/50
+[gen] wrote /root/Adhi/BluTrain/dist/Context_Parallelism/longeval_yarnfix_8192/gen/base_niah_single.csv

logs/longeval_yarnfix_8192/master.log

@@ -0,0 +1,42 @@
+[13:44:18] longeval: T=16384 prompt_len=8192 max_new=24 n=50 greedy=1 gpus=0,1,2,3
+[13:44:18] haystack: Data_Loader/Data/flux_val_000000.bin
+[13:44:18] prompt signature changed -- regenerating prompts and clearing state
+[13:44:18] generating prompts: niah_single
+  niah_single      50 x 8192 tok -> longeval_yarnfix_8192/prompts/niah_single_prompts.bin
+[13:44:19] fan-out: 4 GPU(s), NP=1 -> 4 concurrent job(s)
+[13:44:19] 1 job(s) to run
+[13:44:19]   [gpu 0] base / niah_single
+[13:46:18]   scoring base / niah_single
+
+=== niah_single  (prompt_len=8192, n=50) ===
+  accuracy: 18/50 = 36.0%
+    depth      acc   (needle position in context)
+     0.00   0/10    0.0%
+     0.25   2/10   20.0%
+     0.50   1/10   10.0%
+     0.75   5/10   50.0%
+     1.00  10/10  100.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the special magic number for the yarrow.\n
+  [MISS] expect=5343902
+         got= the number of the cactus that is the special magic number for cactus. The number is determined by the number of
+
+  wrote longeval_yarnfix_8192/scores/base_niah_single.csv
+[13:46:18] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                   base
+  niah_single            36.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth               base
+  0.00           0.0+-13.9
+  0.25          20.0+-22.7
+  0.50          10.0+-19.3
+  0.75          50.0+-26.3
+  1.00         100.0+-13.9
+
+  wrote longeval_yarnfix_8192/summary.csv and longeval_yarnfix_8192/depth_summary.csv
+[13:46:18] DONE -> longeval_yarnfix_8192/summary.csv

logs/longeval_yarnfix_8192/prompts/.signature

@@ -0,0 +1 @@
+N=50|LEN=8192|DEPTHS=0.0,0.25,0.5,0.75,1.0|SEED=0|HAY=flux_val_000000.bin|HAYSZ=76

logs/longeval_yarnfix_8192/prompts/niah_single_meta.json

@@ -0,0 +1,408 @@
+{
+ "task": "niah_single",
+ "prompt_len": 8192,
+ "n": 50,
+ "seed": 0,
+ "items": [
+  {
+   "i": 0,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "8056020"
+   ]
+  },
+  {
+   "i": 1,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "5343902"
+   ]
+  },
+  {
+   "i": 2,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "9152513"
+   ]
+  },
+  {
+   "i": 3,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6088743"
+   ]
+  },
+  {
+   "i": 4,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "7007071"
+   ]
+  },
+  {
+   "i": 5,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9467240"
+   ]
+  },
+  {
+   "i": 6,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "5728454"
+   ]
+  },
+  {
+   "i": 7,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "2590996"
+   ]
+  },
+  {
+   "i": 8,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "9934935"
+   ]
+  },
+  {
+   "i": 9,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6203412"
+   ]
+  },
+  {
+   "i": 10,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "2237192"
+   ]
+  },
+  {
+   "i": 11,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "8921240"
+   ]
+  },
+  {
+   "i": 12,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "6935633"
+   ]
+  },
+  {
+   "i": 13,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6304900"
+   ]
+  },
+  {
+   "i": 14,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "9002896"
+   ]
+  },
+  {
+   "i": 15,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9746862"
+   ]
+  },
+  {
+   "i": 16,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2044878"
+   ]
+  },
+  {
+   "i": 17,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "2564842"
+   ]
+  },
+  {
+   "i": 18,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1019173"
+   ]
+  },
+  {
+   "i": 19,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6589080"
+   ]
+  },
+  {
+   "i": 20,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "6456023"
+   ]
+  },
+  {
+   "i": 21,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "4205573"
+   ]
+  },
+  {
+   "i": 22,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5003310"
+   ]
+  },
+  {
+   "i": 23,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "8515682"
+   ]
+  },
+  {
+   "i": 24,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "2349656"
+   ]
+  },
+  {
+   "i": 25,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "9521829"
+   ]
+  },
+  {
+   "i": 26,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2829687"
+   ]
+  },
+  {
+   "i": 27,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5883691"
+   ]
+  },
+  {
+   "i": 28,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "6582627"
+   ]
+  },
+  {
+   "i": 29,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "4409161"
+   ]
+  },
+  {
+   "i": 30,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "8465677"
+   ]
+  },
+  {
+   "i": 31,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "7457569"
+   ]
+  },
+  {
+   "i": 32,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "5061884"
+   ]
+  },
+  {
+   "i": 33,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "4084805"
+   ]
+  },
+  {
+   "i": 34,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "4132801"
+   ]
+  },
+  {
+   "i": 35,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "5363019"
+   ]
+  },
+  {
+   "i": 36,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "2159058"
+   ]
+  },
+  {
+   "i": 37,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "3184803"
+   ]
+  },
+  {
+   "i": 38,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1648297"
+   ]
+  },
+  {
+   "i": 39,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "7564858"
+   ]
+  },
+  {
+   "i": 40,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "5624373"
+   ]
+  },
+  {
+   "i": 41,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "4951063"
+   ]
+  },
+  {
+   "i": 42,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "8036754"
+   ]
+  },
+  {
+   "i": 43,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "8559194"
+   ]
+  },
+  {
+   "i": 44,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "6995632"
+   ]
+  },
+  {
+   "i": 45,
+   "depth": 0.0,
+   "depth_requested": 0.0,
+   "expected": [
+    "6441267"
+   ]
+  },
+  {
+   "i": 46,
+   "depth": 0.25,
+   "depth_requested": 0.25,
+   "expected": [
+    "9161179"
+   ]
+  },
+  {
+   "i": 47,
+   "depth": 0.5,
+   "depth_requested": 0.5,
+   "expected": [
+    "4194027"
+   ]
+  },
+  {
+   "i": 48,
+   "depth": 0.75,
+   "depth_requested": 0.75,
+   "expected": [
+    "1271942"
+   ]
+  },
+  {
+   "i": 49,
+   "depth": 1.0,
+   "depth_requested": 1.0,
+   "expected": [
+    "2965187"
+   ]
+  }
+ ]
+}

logs/longeval_yarnfix_8192/prompts/niah_single_prompts.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d43f5c59c15e3e8091e493cd6b08ca7b0f9084f457cc056a215dac1dc9d098f2
+size 1638400

logs/longeval_yarnfix_8192/scores/base_niah_single.csv

@@ -0,0 +1,51 @@
+i,depth,ok,n_expected,n_hit,expected,generated
+0,0.0,0,1,0,8056020, the special magic number for the yarrow.\n
+1,0.25,0,1,0,5343902, the number of the cactus that is the special magic number for cactus. The number is determined by the number of
+2,0.5,0,1,0,9152513," the number of the ""stone"" in the granite.\n"
+3,0.75,1,1,1,6088743, 6088743.\n
+4,1.0,1,1,1,7007071, 7007071.\n
+5,0.0,0,1,0,9467240, 7.\nQuestion: What is the significance of the nutmeg number 7?\n
+6,0.25,0,1,0,5728454, the number of the largest iceberg in the world.\n
+7,0.5,0,1,0,2590996, the unique number of the iceberg that makes up the ocean. It’s measured in kilohertz (kHz
+8,0.75,0,1,0,9934935, 7. The quartz crystal is clear and free from any impurities. The quartz crystal is a type of quartz crystal that
+9,1.0,1,1,1,6203412, 6203412.\n
+10,0.0,0,1,0,2237192,", is the glacier that’s the most important thing in the world. Right? It’s the most"
+11,0.25,0,1,0,8921240, the number of violet leaves. This number is crucial for understanding the color and its significance in various cultures.\n
+12,0.5,0,1,0,6935633, the “Paperless Office”.\n
+13,0.75,0,1,0,6304900,":\n(x,y,z) = bramble\n"
+14,1.0,1,1,1,9002896, 9002896. \n
+15,0.0,0,1,0,9746862," the ""Cinder-Cinder-Cinder-Cinder-Cinder-Cinder-Cinder-C"
+16,0.25,0,1,0,2044878," the quartz crystal. Quartz is a very hard, very brittle, quartz crystal. Quartz is a very hard, very brittle"
+17,0.5,0,1,0,2564842," the ""magical"" number of the ""magical"" (or ""magical"") position of the ""mag"
+18,0.75,1,1,1,1019173, 1019173.\n
+19,1.0,1,1,1,6589080, 6589080. \n
+20,0.0,0,1,0,6456023, the special magic number for penguin.\n
+21,0.25,0,1,0,4205573,:\n(1) the mean wave drag force (aka drift) force is computed from the mean wave drag force (
+22,0.5,0,1,0,5003310," the ""low-frequency"" (LF) *variations* in the ""legacy"" normal mode analysis."
+23,0.75,0,1,0,8515682,:\nQuestion: What is the special magic number for jaguar?\n
+24,1.0,1,1,1,2349656, 2349656. \n
+25,0.0,0,1,0,9521829, the special magic number for the umbrella. The umbrella is a special magic number for the umbrella. The umbrella is a special
+26,0.25,0,1,0,2829687," the [mean wave drift force, RMS surge] produced by wave drift forces from a single *single* swell system"
+27,0.5,0,1,0,5883691," the ""Tundra-Tundra-Tundra-Tundra-Tundra-Tund"
+28,0.75,1,1,1,6582627, 6582627.\n
+29,1.0,1,1,1,4409161, 4409161.\n
+30,0.0,0,1,0,8465677, 7.\nHow can I use the Brigganaes Which Light in my life?\n
+31,0.25,0,1,0,7457569, the special magic number for falcon. It is the special magic number for falcon.\n
+32,0.5,0,1,0,5061884,:\n• Ca2+ (Fe(III)):\n
+33,0.75,0,1,0,4084805, Fe(OH)₃. This is the most abundant and stable iron in the soil.\n
+34,1.0,1,1,1,4132801, 4132801. \n
+35,0.0,0,1,0,5363019, typically 0.5-1.0.\n
+36,0.25,0,1,0,2159058, the special magic number for ember. This number is used to identify ember and determine their potential for success in the
+37,0.5,1,1,1,3184803, 3184803.\n
+38,0.75,1,1,1,1648297, 1648297.\n
+39,1.0,1,1,1,7564858, 7564858.\n
+40,0.0,0,1,0,5624373, the number of pockets in the brownie base that make it so gooey and spreadable. It’s
+41,0.25,1,1,1,4951063, 4951063. This number is significant because it indicates that the product is designed to fit the body's contours
+42,0.5,0,1,0,8036754," 9. It's a number that appears in the dreamscape, often appearing in the form of a nutmeg leaf."
+43,0.75,1,1,1,8559194, 8559194.\n
+44,1.0,1,1,1,6995632, 6995632.\n
+45,0.0,0,1,0,6441267, the number of falcones that have been found in the area. The number of falcones that have been found
+46,0.25,1,1,1,9161179, 9161179.\n
+47,0.5,0,1,0,4194027, the number of the first letter of the alphabet.\n
+48,0.75,0,1,0,1271942, the number of penguin that the penguin is able to see. The number is determined by the number of penguins
+49,1.0,1,1,1,2965187, 2965187.\n

logs/longeval_yarnfix_8192/summary.csv

@@ -0,0 +1,2 @@
+task,base
+niah_single,36.0