[08:09:12] longeval: T=16384 prompt_len=4096 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[08:09:12] haystack: Data_Loader/Data/flux_val_000000.bin
[08:09:12] prompt signature changed -- regenerating prompts and clearing state
[08:09:12] generating prompts: niah_single
[08:09:12] generating prompts: val_backward
[08:09:12] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[08:09:12] 4 job(s) to run
[08:09:12]   [gpu 0] base_c / niah_single
[08:09:12]   [gpu 1] base_c / val_backward
[08:09:12]   [gpu 2] armB_c / niah_single
[08:09:12]   [gpu 3] armB_c / val_backward
[08:14:18]   scoring base_c / niah_single

=== niah_single  (prompt_len=4096, n=100) ===
  accuracy: 71/100 = 71.0%
    depth      acc   (needle position in context)
     0.00   9/20   45.0%
     0.25  11/20   55.0%
     0.50  14/20   70.0%
     0.75  17/20   85.0%
     1.00  20/20  100.0%

  --- 2 sample generations ---
  [MISS] expect=8056020
         got= 8.\nQuestion: How do I find a reliable bus rental service?\n
  [MISS] expect=5343902
         got= cactus.\n

  wrote newck_len_4096/scores/base_c_niah_single.csv
[08:14:19]   scoring base_c / val_backward

=== val_backward  (prompt_len=4096, n=100) ===
  accuracy: 4/100 = 4.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   0/20    0.0%
     1.00   4/20   20.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the Magic Number 8056020. It is a unique number that is used to unlock special abilities in the Magic Number
  [MISS] expect=cactus
         got= the 5343902.\n

  wrote newck_len_4096/scores/base_c_val_backward.csv
[08:14:19]   scoring armB_c / niah_single

=== niah_single  (prompt_len=4096, n=100) ===
  accuracy: 60/100 = 60.0%
    depth      acc   (needle position in context)
     0.00   7/20   35.0%
     0.25   8/20   40.0%
     0.50  10/20   50.0%
     0.75  16/20   80.0%
     1.00  19/20   95.0%

  --- 2 sample generations ---
  [MISS] expect=8056020
         got= 8.\nQuestion: How do I find the special magic number for yarrow?\n
  [MISS] expect=5343902
         got= cactus.\n

  wrote newck_len_4096/scores/armB_c_niah_single.csv
[08:14:19]   scoring armB_c / val_backward

=== val_backward  (prompt_len=4096, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   0/20    0.0%
     1.00   1/20    5.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the number 8056020.\n
  [MISS] expect=cactus
         got= the Bitcoin lottery.\n

  wrote newck_len_4096/scores/armB_c_val_backward.csv
[08:14:19] building summary

=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
  task                 armB_c   base_c
  niah_single            60.0     71.0
  val_backward            1.0      4.0

=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
  depth             armB_c          base_c
  0.00          17.5+-11.6      22.5+-12.6
  0.25          20.0+-12.1      27.5+-13.4
  0.50          25.0+-13.0      35.0+-14.2
  0.75          40.0+-14.5      42.5+-14.6
  1.00          50.0+-14.8      60.0+-14.5

  wrote newck_len_4096/summary.csv and newck_len_4096/depth_summary.csv
[08:14:19] DONE -> newck_len_4096/summary.csv