Upload logs/longeval_16k/master_run.log with huggingface_hub

logs/longeval_16k/master_run.log

@@ -0,0 +1,401 @@
+[11:33:38] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3
+[11:33:38] haystack: Data_Loader/Data/flux_val_000000.bin
+[11:33:38] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[11:33:38]   base/niah_single already generated
+[11:33:38]   base/niah_multikey already generated
+[11:33:38]   base/niah_multivalue already generated
+[11:33:38]   base/vt already generated
+[11:33:38]   base/val_backward already generated
+[11:33:38]   arm_a/niah_single already generated
+[11:33:38]   arm_a/niah_multikey already generated
+[11:33:38] 13 job(s) to run
+[11:33:38]   [gpu 0] arm_a / niah_multivalue
+[11:33:38]   [gpu 1] arm_a / vt
+[11:33:38]   [gpu 2] arm_a / val_backward
+[11:33:38]   [gpu 3] arm_b / niah_single
+[11:33:38]   [gpu 0] arm_b / niah_multikey
+[11:33:38]   [gpu 1] arm_b / niah_multivalue
+[11:33:38]   [gpu 2] arm_b / vt
+[11:33:38]   [gpu 3] arm_b / val_backward
+[11:54:02]   [gpu 0] arm_c / niah_single
+[11:54:15]   [gpu 1] arm_c / niah_multikey
+[11:55:52]   [gpu 2] arm_c / niah_multivalue
+[11:56:31]   [gpu 3] arm_c / vt
+[11:56:41]   [gpu 0] arm_c / val_backward
+[12:16:44]   scoring base / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got=:\nQuestion: What is the Devil?\n
+  [MISS] expect=5343902
+         got= not known, what the period modes: transportation of the cactus?\n
+
+  wrote longeval_runs/scores/base_niah_single.csv
+[12:16:44]   scoring base / niah_multikey
+
+=== niah_multikey  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=6088743
+         got= the first, and the magic number is made by the same.\n
+  [MISS] expect=9934935
+         got= the following:\n
+
+  wrote longeval_runs/scores/base_niah_multikey.csv
+[12:16:44]   scoring base / niah_multivalue
+
+=== niah_multivalue  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020|1679215|5343902
+         got= the magic, from zero-time, the magic.\nQuestion: What is the wild?\n
+  [MISS] expect=8995970|7007071|4664860
+         got= the same thing.\nWhere does the religious events of the same age are tundra?\n
+
+  wrote longeval_runs/scores/base_niah_multivalue.csv
+[12:16:44]   scoring base / vt
+
+=== vt  (prompt_len=16360, n=100) ===
+  accuracy: 1/100 = 1.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     0.80   1/20    5.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=5370335
+         got= “VAR”, “California”, “California night” is a perfect setting.
+  [MISS] expect=6319163
+         got= VAR TV, VAR is not just like VAR.\n
+
+  wrote longeval_runs/scores/base_vt.csv
+[12:16:45]   scoring base / val_backward
+
+=== val_backward  (prompt_len=16360, n=100) ===
+  accuracy: 1/100 = 1.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   1/20    5.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= a magical, but we use 80560 to make a magical, open and peaceful. You may get the magic of
+  [MISS] expect=cactus
+         got= the special lifestyle.\n
+
+  wrote longeval_runs/scores/base_val_backward.csv
+[12:16:45]   scoring arm_a / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the same.\n
+  [MISS] expect=5343902
+         got= the special of the cactus.\n
+
+  wrote longeval_runs/scores/arm_a_niah_single.csv
+[12:16:45]   scoring arm_a / niah_multikey
+
+=== niah_multikey  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=6088743
+         got= the same.\n
+  [MISS] expect=9934935
+         got= the same for the car.\n
+
+  wrote longeval_runs/scores/arm_a_niah_multikey.csv
+[12:16:45]   scoring arm_a / niah_multivalue
+
+=== niah_multivalue  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020|1679215|5343902
+         got= the same.\nQuestion: Juan Rodriguez Cabrillo\nNorto Cabrillo\n
+  [MISS] expect=8995970|7007071|4664860
+         got= the same for tundra.\nQuestion: The other tundra is the same of the tundra\n
+
+  wrote longeval_runs/scores/arm_a_niah_multivalue.csv
+[12:16:45]   scoring arm_a / vt
+
+=== vt  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     0.80   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=5370335
+         got=\n“VAR HOL88,” the VAR, and the VAR, and the VAR,
+  [MISS] expect=6319163
+         got=\nVAR YAR\n
+
+  wrote longeval_runs/scores/arm_a_vt.csv
+[12:16:46]   scoring arm_a / val_backward
+
+=== val_backward  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the one of the the same.\n
+  [MISS] expect=cactus
+         got= the\nHow many of the special, the same of thin old grey stone Georgian mansion is the\n
+
+  wrote longeval_runs/scores/arm_a_val_backward.csv
+[12:16:46]   scoring arm_b / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the, and the only fight is the same.\n
+  [MISS] expect=5343902
+         got= the.\nDo the language cactus, that is the other.\n
+
+  wrote longeval_runs/scores/arm_b_niah_single.csv
+[12:16:46]   scoring arm_b / niah_multikey
+
+=== niah_multikey  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=6088743
+         got= the same. Steam was the stone in the C.\n
+  [MISS] expect=9934935
+         got= the\nHow rumorian\n
+
+  wrote longeval_runs/scores/arm_b_niah_multikey.csv
+[12:16:46]   scoring arm_b / niah_multivalue
+
+=== niah_multivalue  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020|1679215|5343902
+         got= the.\nThe name silk in the e-Crookeeng, the<|endoftext|>Possar, and the name of
+  [MISS] expect=8995970|7007071|4664860
+         got= the.\nPhrenoettes of the same\nTermany measurements of the same\n
+
+  wrote longeval_runs/scores/arm_b_niah_multivalue.csv
+[12:16:46]   scoring arm_b / vt
+
+=== vt  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     0.80   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=5370335
+         got= a lot of the experience.\n
+  [MISS] expect=6319163
+         got=\nDebt the language of VAR. The Cock House of the same\n
+
+  wrote longeval_runs/scores/arm_b_vt.csv
+[12:16:47]   scoring arm_b / val_backward
+
+=== val_backward  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the most.\n
+  [MISS] expect=cactus
+         got=,\nPropver Indian Old Women\n
+
+  wrote longeval_runs/scores/arm_b_val_backward.csv
+[12:16:47]   scoring arm_c / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the same, the best time.\n
+  [MISS] expect=5343902
+         got= the\nFirst of the book\n
+
+  wrote longeval_runs/scores/arm_c_niah_single.csv
+[12:16:47]   scoring arm_c / niah_multikey
+
+=== niah_multikey  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=6088743
+         got= the\nThe first time of the Cabrillo\n
+  [MISS] expect=9934935
+         got= a time.\n
+
+  wrote longeval_runs/scores/arm_c_niah_multikey.csv
+[12:16:47]   scoring arm_c / niah_multivalue
+
+=== niah_multivalue  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020|1679215|5343902
+         got= the first time.\nThe timberline in the first of the world, the the first of the world, the the
+  [MISS] expect=8995970|7007071|4664860
+         got= the way!\nIs the way a tundra\nIf you are the tundra\n
+
+  wrote longeval_runs/scores/arm_c_niah_multivalue.csv
+[12:16:47]   scoring arm_c / vt
+
+=== vt  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     0.80   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=5370335
+         got= the first time\n
+  [MISS] expect=6319163
+         got= V-!\n
+
+  wrote longeval_runs/scores/arm_c_vt.csv
+[12:16:48]   scoring arm_c / val_backward
+
+=== val_backward  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=yarrow
+         got= the same.\n
+  [MISS] expect=cactus
+         got= the.\nQuestion:\n
+
+  wrote longeval_runs/scores/arm_c_val_backward.csv
+[12:16:48] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                  arm_a    arm_b    arm_c     base
+  niah_multikey           0.0      0.0      0.0      0.0
+  niah_multivalue         0.0      0.0      0.0      0.0
+  niah_single             0.0      0.0      0.0      0.0
+  val_backward            0.0      0.0      0.0      1.0
+  vt                      0.0      0.0      0.0      1.0
+
+  EXCLUDED from pooling (<=1% for every model): ['niah_multikey', 'niah_multivalue', 'niah_single', 'val_backward', 'vt']
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth              arm_a           arm_b           arm_c            base
+
+  wrote longeval_runs/summary.csv and longeval_runs/depth_summary.csv
+[12:16:48] DONE -> longeval_runs/summary.csv