[08:14:19] longeval: T=16384 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[08:14:19] haystack: Data_Loader/Data/flux_val_000000.bin
[08:14:19] prompt signature changed -- regenerating prompts and clearing state
[08:14:19] generating prompts: niah_single
[08:14:20] generating prompts: val_backward
[08:14:20] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[08:14:20] 4 job(s) to run
[08:14:20]   [gpu 0] base_c / niah_single
[08:14:20]   [gpu 1] base_c / val_backward
[08:14:20]   [gpu 2] armB_c / niah_single
[08:14:20]   [gpu 3] armB_c / val_backward
[08:22:43]   scoring base_c / niah_single

=== niah_single  (prompt_len=8192, n=100) ===
  accuracy: 38/100 = 38.0%
    depth      acc   (needle position in context)
     0.00   3/20   15.0%
     0.25   2/20   10.0%
     0.50   5/20   25.0%
     0.75   9/20   45.0%
     1.00  19/20   95.0%

  --- 2 sample generations ---
  [MISS] expect=8056020
         got= 0.\nunimg[asciilifeform]: I'm not sure what it is, but I think it
  [MISS] expect=5343902
         got= the number of the cactus’s eyes.\n

  wrote newck_len_8192/scores/base_c_niah_single.csv
[08:22:43]   scoring base_c / val_backward

=== val_backward  (prompt_len=8192, n=100) ===
  accuracy: 4/100 = 4.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   1/20    5.0%
     1.00   3/20   15.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the first item in the "Chans" series.\n
  [MISS] expect=cactus
         got= the number 5343902.\n

  wrote newck_len_8192/scores/base_c_val_backward.csv
[08:22:44]   scoring armB_c / niah_single

=== niah_single  (prompt_len=8192, n=100) ===
  accuracy: 31/100 = 31.0%
    depth      acc   (needle position in context)
     0.00   3/20   15.0%
     0.25   2/20   10.0%
     0.50   4/20   20.0%
     0.75   7/20   35.0%
     1.00  15/20   75.0%

  --- 2 sample generations ---
  [MISS] expect=8056020
         got= 1.\nunpx[asciilifeform]: I'm not sure what it is, but I'm not
  [MISS] expect=5343902
         got= the number of the plant’s leaves.\n

  wrote newck_len_8192/scores/armB_c_niah_single.csv
[08:22:44]   scoring armB_c / val_backward

=== val_backward  (prompt_len=8192, n=100) ===
  accuracy: 2/100 = 2.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   0/20    0.0%
     1.00   2/20   10.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the one that has the special magic number 8056020.\n
  [MISS] expect=cactus
         got= the number 5343902.\n

  wrote newck_len_8192/scores/armB_c_val_backward.csv
[08:22:44] building summary

=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
  task                 armB_c   base_c
  niah_single            31.0     38.0
  val_backward            2.0      4.0

=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
  depth             armB_c          base_c
  0.00           7.5+- 8.6       7.5+- 8.6
  0.25           5.0+- 7.6       5.0+- 7.6
  0.50          10.0+- 9.5      12.5+-10.3
  0.75          17.5+-11.6      25.0+-13.0
  1.00          42.5+-14.6      55.0+-14.7

  wrote newck_len_8192/summary.csv and newck_len_8192/depth_summary.csv
[08:22:44] DONE -> newck_len_8192/summary.csv