Upload logs/longeval_fix16k/driver.log with huggingface_hub

logs/longeval_fix16k/driver.log

@@ -0,0 +1,177 @@
+[13:52:09] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3
+[13:52:09] haystack: Data_Loader/Data/flux_val_000000.bin
+[13:52:09] prompt signature changed -- regenerating prompts and clearing state
+[13:52:09] generating prompts: niah_single
+[13:52:10] generating prompts: vt
+[13:52:10] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
+[13:52:10] 8 job(s) to run
+[13:52:10]   [gpu 0] base / niah_single
+[13:52:10]   [gpu 1] base / vt
+[13:52:10]   [gpu 2] arm_a / niah_single
+[13:52:10]   [gpu 3] arm_a / vt
+[13:52:10]   [gpu 0] arm_b / niah_single
+[13:52:10]   [gpu 1] arm_b / vt
+[13:52:10]   [gpu 2] arm_c / niah_single
+[13:52:10]   [gpu 3] arm_c / vt
+[14:23:46]   scoring base / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 17/100 = 17.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   5/20   25.0%
+     1.00  12/20   60.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the number of the elm, which is the elm of the elm of the elm of the elm
+  [MISS] expect=5343902
+         got= the number of cactus.\n
+
+  wrote longeval_fix16k/scores/base_niah_single.csv
+[14:23:46]   scoring base / vt
+
+=== vt  (prompt_len=16360, n=100) ===
+  accuracy: 4/100 = 4.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   2/20   10.0%
+     0.80   2/20   10.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=5370335
+         got= VAR HOL88\n
+  [MISS] expect=6319163
+         got= VAR YAR75 = VAR YAR75 = VAR YAR75 = VAR YAR75 =
+
+  wrote longeval_fix16k/scores/base_vt.csv
+[14:23:46]   scoring arm_a / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 17/100 = 17.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   4/20   20.0%
+     1.00  13/20   65.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the number of yarrow’s magic. The number of yarrow’s magic is the number of y
+  [MISS] expect=5343902
+         got= the special magic number.\n
+
+  wrote longeval_fix16k/scores/arm_a_niah_single.csv
+[14:23:46]   scoring arm_a / vt
+
+=== vt  (prompt_len=16360, n=100) ===
+  accuracy: 5/100 = 5.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   2/20   10.0%
+     0.80   3/20   15.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=5370335
+         got= VAR HOL88.\n
+  [MISS] expect=6319163
+         got= VAR YAR75\n
+
+  wrote longeval_fix16k/scores/arm_a_vt.csv
+[14:23:47]   scoring arm_b / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 4/100 = 4.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00   4/20   20.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the most important. The magic number is the most important. The magic number is the most important. The magic number is
+  [MISS] expect=5343902
+         got= the number of the magic number.\n
+
+  wrote longeval_fix16k/scores/arm_b_niah_single.csv
+[14:23:47]   scoring arm_b / vt
+
+=== vt  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     0.80   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=5370335
+         got= the VAR.\n
+  [MISS] expect=6319163
+         got= VAR YAR75\n
+
+  wrote longeval_fix16k/scores/arm_b_vt.csv
+[14:23:47]   scoring arm_c / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 18/100 = 18.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   1/20    5.0%
+     0.75   6/20   30.0%
+     1.00  11/20   55.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the special magic number for yarrow.\n
+  [MISS] expect=5343902
+         got= the special magic number for the cactus.\n
+
+  wrote longeval_fix16k/scores/arm_c_niah_single.csv
+[14:23:47]   scoring arm_c / vt
+
+=== vt  (prompt_len=16360, n=100) ===
+  accuracy: 0/100 = 0.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     0.80   0/20    0.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=5370335
+         got= VAR HOL88.\n
+  [MISS] expect=6319163
+         got= VAR YAR75 = VAR YAR75 = VAR YAR75\n
+
+  wrote longeval_fix16k/scores/arm_c_vt.csv
+[14:23:47] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                  arm_a    arm_b    arm_c     base
+  niah_single            17.0      4.0     18.0     17.0
+  vt                      5.0      0.0      0.0      4.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth              arm_a           arm_b           arm_c            base
+  0.00           0.0+- 4.4       0.0+- 4.4       0.0+- 4.4       0.0+- 4.4
+  0.25           0.0+- 4.4       0.0+- 4.4       0.0+- 4.4       0.0+- 4.4
+  0.50           0.0+- 4.4       0.0+- 4.4       2.5+- 6.2       0.0+- 4.4
+  0.75          15.0+-11.0       0.0+- 4.4      15.0+-11.0      17.5+-11.6
+  0.80          15.0+-15.4       0.0+- 8.1       0.0+- 8.1      10.0+-13.7
+  1.00          65.0+-19.3      20.0+-16.8      55.0+-20.0      60.0+-19.7
+
+  wrote longeval_fix16k/summary.csv and longeval_fix16k/depth_summary.csv
+[14:23:47] DONE -> longeval_fix16k/summary.csv