Upload folder using huggingface_hub

logs/eval_run7c/eval_1024.log

@@ -0,0 +1,100 @@
+[08:01:42] longeval: T=16384 prompt_len=1024 max_new=24 n=100 greedy=1 gpus=0,1,2,3
+[08:01:42] haystack: Data_Loader/Data/flux_val_000000.bin
+[08:01:42] prompt signature changed -- regenerating prompts and clearing state
+[08:01:42] generating prompts: niah_single
+[08:01:43] generating prompts: val_backward
+[08:01:43] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[08:01:43] 4 job(s) to run
+[08:01:43]   [gpu 0] base_c / niah_single
+[08:01:43]   [gpu 1] base_c / val_backward
+[08:01:43]   [gpu 2] armB_c / niah_single
+[08:01:43]   [gpu 3] armB_c / val_backward
+[08:05:14]   scoring base_c / niah_single
+
+=== niah_single  (prompt_len=1024, n=100) ===
+  accuracy: 87/100 = 87.0%
+    depth      acc   (needle position in context)
+     0.00  17/20   85.0%
+     0.25  16/20   80.0%
+     0.50  16/20   80.0%
+     0.75  18/20   90.0%
+     1.00  20/20  100.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020\n
+  [MISS] expect=5343902
+         got= 7.\nScum (1979 - film version)\n
+
+  wrote newck_len_1024/scores/base_c_niah_single.csv
+[08:05:14]   scoring base_c / val_backward
+
+=== val_backward  (prompt_len=1024, n=100) ===
+  accuracy: 3/100 = 3.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   3/20   15.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the...\n3. I’m not sure what the special magic number 8056020 is, but...
+  [MISS] expect=cactus
+         got= a\nscum (1979 - film version)\n
+
+  wrote newck_len_1024/scores/base_c_val_backward.csv
+[08:05:15]   scoring armB_c / niah_single
+
+=== niah_single  (prompt_len=1024, n=100) ===
+  accuracy: 87/100 = 87.0%
+    depth      acc   (needle position in context)
+     0.00  17/20   85.0%
+     0.25  16/20   80.0%
+     0.50  16/20   80.0%
+     0.75  18/20   90.0%
+     1.00  20/20  100.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020\n
+  [MISS] expect=5343902
+         got= 7.\nThe special magic number for cactus is 7.\n
+
+  wrote newck_len_1024/scores/armB_c_niah_single.csv
+[08:05:15]   scoring armB_c / val_backward
+
+=== val_backward  (prompt_len=1024, n=100) ===
+  accuracy: 2/100 = 2.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   1/20    5.0%
+     0.75   0/20    0.0%
+     1.00   1/20    5.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the\n3. I’m not sure if Moomin is a fan of the game, but I�
+  [MISS] expect=cactus
+         got= the\nscum (1979 - film version)\n
+
+  wrote newck_len_1024/scores/armB_c_val_backward.csv
+[08:05:15] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 armB_c   base_c
+  niah_single            87.0     87.0
+  val_backward            2.0      3.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             armB_c          base_c
+  0.00          42.5+-14.6      42.5+-14.6
+  0.25          40.0+-14.5      40.0+-14.5
+  0.50          42.5+-14.6      40.0+-14.5
+  0.75          45.0+-14.7      45.0+-14.7
+  1.00          52.5+-14.8      57.5+-14.6
+
+  wrote newck_len_1024/summary.csv and newck_len_1024/depth_summary.csv
+[08:05:15] DONE -> newck_len_1024/summary.csv

logs/eval_run7c/eval_16360.log

@@ -0,0 +1,102 @@
+[08:22:44] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3
+[08:22:44] haystack: Data_Loader/Data/flux_val_000000.bin
+[08:22:44] prompt signature changed -- regenerating prompts and clearing state
+[08:22:44] generating prompts: niah_single
+[08:22:45] generating prompts: val_backward
+[08:22:45] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[08:22:45] 4 job(s) to run
+[08:22:45]   [gpu 0] base_c / niah_single
+[08:22:45]   [gpu 1] base_c / val_backward
+[08:22:45]   [gpu 2] armB_c / niah_single
+[08:22:45]   [gpu 3] armB_c / val_backward
+[08:35:05]   scoring base_c / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 27/100 = 27.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   2/20   10.0%
+     0.75   7/20   35.0%
+     1.00  18/20   90.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the number of the number of the number of the number of the number of the number of the number of the number of
+  [MISS] expect=5343902
+         got= 7.\nHow much does Anna Katarina earn per year?\n
+
+  wrote newck_len_16360/scores/base_c_niah_single.csv
+[08:35:06]   scoring base_c / val_backward
+
+=== val_backward  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the number 8056020.\n
+  [MISS] expect=cactus
+         got= called "The Magician".\n
+
+  wrote newck_len_16360/scores/base_c_val_backward.csv
+[08:35:06]   scoring armB_c / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 23/100 = 23.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   2/20   10.0%
+     0.75   4/20   20.0%
+     1.00  17/20   85.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the number of the moon.\n
+  [MISS] expect=5343902
+         got= 7.\nQuestion: What is the birth name of Anna Katarina?\n
+
+  wrote newck_len_16360/scores/armB_c_niah_single.csv
+[08:35:06]   scoring armB_c / val_backward
+
+=== val_backward  (prompt_len=16360, n=100) ===
+  accuracy: 1/100 = 1.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   1/20    5.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the number 8056020.\n
+  [MISS] expect=cactus
+         got=: "The Magician".\n
+
+  wrote newck_len_16360/scores/armB_c_val_backward.csv
+[08:35:06] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 armB_c   base_c
+  niah_single            23.0     27.0
+  val_backward            1.0      0.0
+
+  degenerate (<=1% for every model): ['val_backward']
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             armB_c          base_c
+  0.00           0.0+- 8.1       0.0+- 8.1
+  0.25           0.0+- 8.1       0.0+- 8.1
+  0.50          10.0+-13.7      10.0+-13.7
+  0.75          20.0+-16.8      35.0+-19.3
+  1.00          85.0+-15.4      90.0+-13.7
+
+  wrote newck_len_16360/summary.csv and newck_len_16360/depth_summary.csv
+[08:35:06] DONE -> newck_len_16360/summary.csv

logs/eval_run7c/eval_2048.log

@@ -0,0 +1,102 @@
+[08:05:15] longeval: T=16384 prompt_len=2048 max_new=24 n=100 greedy=1 gpus=0,1,2,3
+[08:05:15] haystack: Data_Loader/Data/flux_val_000000.bin
+[08:05:15] prompt signature changed -- regenerating prompts and clearing state
+[08:05:15] generating prompts: niah_single
+[08:05:15] generating prompts: val_backward
+[08:05:16] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[08:05:16] 4 job(s) to run
+[08:05:16]   [gpu 0] base_c / niah_single
+[08:05:16]   [gpu 1] base_c / val_backward
+[08:05:16]   [gpu 2] armB_c / niah_single
+[08:05:16]   [gpu 3] armB_c / val_backward
+[08:09:10]   scoring base_c / niah_single
+
+=== niah_single  (prompt_len=2048, n=100) ===
+  accuracy: 82/100 = 82.0%
+    depth      acc   (needle position in context)
+     0.00  11/20   55.0%
+     0.25  14/20   70.0%
+     0.50  19/20   95.0%
+     0.75  19/20   95.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 7.\nScum (1979 - film version)\n
+  [MISS] expect=5343902
+         got= 3.\nQuestion: How do I find a reliable cactus rental service?\n
+
+  wrote newck_len_2048/scores/base_c_niah_single.csv
+[08:09:11]   scoring base_c / val_backward
+
+=== val_backward  (prompt_len=2048, n=100) ===
+  accuracy: 1/100 = 1.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   1/20    5.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the\nscum (1979 - film version)\n
+  [MISS] expect=cactus
+         got= the Magic Wheel. It is a unique and powerful magic number that can transform any ordinary object into a magical force.\n
+
+  wrote newck_len_2048/scores/base_c_val_backward.csv
+[08:09:11]   scoring armB_c / niah_single
+
+=== niah_single  (prompt_len=2048, n=100) ===
+  accuracy: 73/100 = 73.0%
+    depth      acc   (needle position in context)
+     0.00   8/20   40.0%
+     0.25  15/20   75.0%
+     0.50  18/20   90.0%
+     0.75  16/20   80.0%
+     1.00  16/20   80.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 7.\nThe special magic number for yarrow is 7.\n
+  [MISS] expect=5343902
+         got= 2.\nQuestion: How does the magic number for cactus work?\n
+
+  wrote newck_len_2048/scores/armB_c_niah_single.csv
+[08:09:11]   scoring armB_c / val_backward
+
+=== val_backward  (prompt_len=2048, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the\nscum (1979 - film version)\n
+  [MISS] expect=cactus
+         got= the Magic Number 5343902.\n
+
+  wrote newck_len_2048/scores/armB_c_val_backward.csv
+[08:09:11] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 armB_c   base_c
+  niah_single            73.0     82.0
+  val_backward            0.0      1.0
+
+  degenerate (<=1% for every model): ['val_backward']
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             armB_c          base_c
+  0.00          40.0+-19.7      55.0+-20.0
+  0.25          75.0+-17.8      70.0+-18.7
+  0.50          90.0+-13.7      95.0+-11.4
+  0.75          80.0+-16.8      95.0+-11.4
+  1.00          80.0+-16.8      95.0+-11.4
+
+  wrote newck_len_2048/summary.csv and newck_len_2048/depth_summary.csv
+[08:09:11] DONE -> newck_len_2048/summary.csv

logs/eval_run7c/eval_4096.log

@@ -0,0 +1,100 @@
+[08:09:12] longeval: T=16384 prompt_len=4096 max_new=24 n=100 greedy=1 gpus=0,1,2,3
+[08:09:12] haystack: Data_Loader/Data/flux_val_000000.bin
+[08:09:12] prompt signature changed -- regenerating prompts and clearing state
+[08:09:12] generating prompts: niah_single
+[08:09:12] generating prompts: val_backward
+[08:09:12] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[08:09:12] 4 job(s) to run
+[08:09:12]   [gpu 0] base_c / niah_single
+[08:09:12]   [gpu 1] base_c / val_backward
+[08:09:12]   [gpu 2] armB_c / niah_single
+[08:09:12]   [gpu 3] armB_c / val_backward
+[08:14:18]   scoring base_c / niah_single
+
+=== niah_single  (prompt_len=4096, n=100) ===
+  accuracy: 71/100 = 71.0%
+    depth      acc   (needle position in context)
+     0.00   9/20   45.0%
+     0.25  11/20   55.0%
+     0.50  14/20   70.0%
+     0.75  17/20   85.0%
+     1.00  20/20  100.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 8.\nQuestion: How do I find a reliable bus rental service?\n
+  [MISS] expect=5343902
+         got= cactus.\n
+
+  wrote newck_len_4096/scores/base_c_niah_single.csv
+[08:14:19]   scoring base_c / val_backward
+
+=== val_backward  (prompt_len=4096, n=100) ===
+  accuracy: 4/100 = 4.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   4/20   20.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the Magic Number 8056020. It is a unique number that is used to unlock special abilities in the Magic Number
+  [MISS] expect=cactus
+         got= the 5343902.\n
+
+  wrote newck_len_4096/scores/base_c_val_backward.csv
+[08:14:19]   scoring armB_c / niah_single
+
+=== niah_single  (prompt_len=4096, n=100) ===
+  accuracy: 60/100 = 60.0%
+    depth      acc   (needle position in context)
+     0.00   7/20   35.0%
+     0.25   8/20   40.0%
+     0.50  10/20   50.0%
+     0.75  16/20   80.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 8.\nQuestion: How do I find the special magic number for yarrow?\n
+  [MISS] expect=5343902
+         got= cactus.\n
+
+  wrote newck_len_4096/scores/armB_c_niah_single.csv
+[08:14:19]   scoring armB_c / val_backward
+
+=== val_backward  (prompt_len=4096, n=100) ===
+  accuracy: 1/100 = 1.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   1/20    5.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the number 8056020.\n
+  [MISS] expect=cactus
+         got= the Bitcoin lottery.\n
+
+  wrote newck_len_4096/scores/armB_c_val_backward.csv
+[08:14:19] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 armB_c   base_c
+  niah_single            60.0     71.0
+  val_backward            1.0      4.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             armB_c          base_c
+  0.00          17.5+-11.6      22.5+-12.6
+  0.25          20.0+-12.1      27.5+-13.4
+  0.50          25.0+-13.0      35.0+-14.2
+  0.75          40.0+-14.5      42.5+-14.6
+  1.00          50.0+-14.8      60.0+-14.5
+
+  wrote newck_len_4096/summary.csv and newck_len_4096/depth_summary.csv
+[08:14:19] DONE -> newck_len_4096/summary.csv

logs/eval_run7c/eval_8192.log

@@ -0,0 +1,100 @@
+[08:14:19] longeval: T=16384 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1,2,3
+[08:14:19] haystack: Data_Loader/Data/flux_val_000000.bin
+[08:14:19] prompt signature changed -- regenerating prompts and clearing state
+[08:14:19] generating prompts: niah_single
+[08:14:20] generating prompts: val_backward
+[08:14:20] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[08:14:20] 4 job(s) to run
+[08:14:20]   [gpu 0] base_c / niah_single
+[08:14:20]   [gpu 1] base_c / val_backward
+[08:14:20]   [gpu 2] armB_c / niah_single
+[08:14:20]   [gpu 3] armB_c / val_backward
+[08:22:43]   scoring base_c / niah_single
+
+=== niah_single  (prompt_len=8192, n=100) ===
+  accuracy: 38/100 = 38.0%
+    depth      acc   (needle position in context)
+     0.00   3/20   15.0%
+     0.25   2/20   10.0%
+     0.50   5/20   25.0%
+     0.75   9/20   45.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 0.\nunimg[asciilifeform]: I'm not sure what it is, but I think it
+  [MISS] expect=5343902
+         got= the number of the cactus’s eyes.\n
+
+  wrote newck_len_8192/scores/base_c_niah_single.csv
+[08:22:43]   scoring base_c / val_backward
+
+=== val_backward  (prompt_len=8192, n=100) ===
+  accuracy: 4/100 = 4.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   1/20    5.0%
+     1.00   3/20   15.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the first item in the "Chans" series.\n
+  [MISS] expect=cactus
+         got= the number 5343902.\n
+
+  wrote newck_len_8192/scores/base_c_val_backward.csv
+[08:22:44]   scoring armB_c / niah_single
+
+=== niah_single  (prompt_len=8192, n=100) ===
+  accuracy: 31/100 = 31.0%
+    depth      acc   (needle position in context)
+     0.00   3/20   15.0%
+     0.25   2/20   10.0%
+     0.50   4/20   20.0%
+     0.75   7/20   35.0%
+     1.00  15/20   75.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 1.\nunpx[asciilifeform]: I'm not sure what it is, but I'm not
+  [MISS] expect=5343902
+         got= the number of the plant’s leaves.\n
+
+  wrote newck_len_8192/scores/armB_c_niah_single.csv
+[08:22:44]   scoring armB_c / val_backward
+
+=== val_backward  (prompt_len=8192, n=100) ===
+  accuracy: 2/100 = 2.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   2/20   10.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the one that has the special magic number 8056020.\n
+  [MISS] expect=cactus
+         got= the number 5343902.\n
+
+  wrote newck_len_8192/scores/armB_c_val_backward.csv
+[08:22:44] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 armB_c   base_c
+  niah_single            31.0     38.0
+  val_backward            2.0      4.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             armB_c          base_c
+  0.00           7.5+- 8.6       7.5+- 8.6
+  0.25           5.0+- 7.6       5.0+- 7.6
+  0.50          10.0+- 9.5      12.5+-10.3
+  0.75          17.5+-11.6      25.0+-13.0
+  1.00          42.5+-14.6      55.0+-14.7
+
+  wrote newck_len_8192/summary.csv and newck_len_8192/depth_summary.csv
+[08:22:44] DONE -> newck_len_8192/summary.csv

logs/eval_run7c/eval_master.log

@@ -0,0 +1,23 @@
+[07:53:36] arm B still training -- waiting for it to finish (not killing it: its
+[07:53:36]   step-200 shard on rank_0 was lost to a run-number collision, so a
+[07:53:36]   resume is impossible and a kill would cost 41 min, not 12)
+[08:01:42] GPUs free
+[08:01:42] registered base_c = blumodelcp_run6_step_750.ckpt  (run7', corrected cache)
+[08:01:42] registered armB_c = blumodelcp_run7_step_400.ckpt  (all 4 shards present)
+[08:01:42] models: base_c armB_c 
+[08:01:42] eval PROMPT_LEN=1024  N=100  tasks: niah_single val_backward
+[08:05:15]   -> 3 rows in newck_len_1024/summary.csv
+[08:05:15] eval PROMPT_LEN=2048  N=100  tasks: niah_single val_backward
+[08:09:11]   -> 3 rows in newck_len_2048/summary.csv
+[08:09:11] eval PROMPT_LEN=4096  N=100  tasks: niah_single val_backward
+[08:14:19]   -> 3 rows in newck_len_4096/summary.csv
+[08:14:19] eval PROMPT_LEN=8192  N=100  tasks: niah_single val_backward
+[08:22:44]   -> 3 rows in newck_len_8192/summary.csv
+[08:22:44] eval PROMPT_LEN=16360  N=100  tasks: niah_single val_backward
+[08:35:06]   -> 3 rows in newck_len_16360/summary.csv
+[08:35:06] EVAL DONE
+[08:35:06]   newck_len_1024/summary.csv OK
+[08:35:06]   newck_len_2048/summary.csv OK
+[08:35:06]   newck_len_4096/summary.csv OK
+[08:35:06]   newck_len_8192/summary.csv OK
+[08:35:06]   newck_len_16360/summary.csv OK