[08:56:22] longeval: T=16384 prompt_len=1024 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[08:56:22] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin
[08:56:22] prompt signature changed -- regenerating prompts and clearing state
[08:56:22] generating prompts: niah_single
  niah_single      100 x 1024 tok -> lenb_len_1024/prompts/niah_single_prompts.bin
[08:56:22] generating prompts: val_backward
  val_backward     100 x 1024 tok -> lenb_len_1024/prompts/val_backward_prompts.bin
[08:56:23] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[08:56:23] 6 job(s) to run
[08:56:23]   [gpu 0] armA_c / niah_single
[08:56:23]   [gpu 1] armA_c / val_backward
[08:56:23]   [gpu 2] l1 / niah_single
[08:56:23]   [gpu 3] l1 / val_backward
[08:56:23]   [gpu 0] l2 / niah_single
[08:56:23]   [gpu 1] l2 / val_backward
[09:01:58]   scoring armA_c / niah_single

=== niah_single  (prompt_len=1024, n=100) ===
  accuracy: 94/100 = 94.0%
    depth      acc   (needle position in context)
     0.00  17/20   85.0%
     0.25  20/20  100.0%
     0.50  19/20   95.0%
     0.75  20/20  100.0%
     1.00  18/20   90.0%

  --- 2 sample generations ---
  [HIT ] expect=8056020
         got= 8056020.\n
  [HIT ] expect=5343902
         got= 5343902.  shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward

  wrote lenb_len_1024/scores/armA_c_niah_single.csv
[09:01:58]   scoring armA_c / val_backward

=== val_backward  (prompt_len=1024, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   1/20    5.0%
     1.00   0/20    0.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the one that has the special magic number 8056020.\n
  [MISS] expect=cactus
         got= the Zangoose.\n

  wrote lenb_len_1024/scores/armA_c_val_backward.csv
[09:01:58]   scoring l1 / niah_single

=== niah_single  (prompt_len=1024, n=100) ===
  accuracy: 95/100 = 95.0%
    depth      acc   (needle position in context)
     0.00  18/20   90.0%
     0.25  20/20  100.0%
     0.50  19/20   95.0%
     0.75  20/20  100.0%
     1.00  18/20   90.0%

  --- 2 sample generations ---
  [HIT ] expect=8056020
         got= 805602020.\n
  [HIT ] expect=5343902
         got= 5343902.  shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward

  wrote lenb_len_1024/scores/l1_niah_single.csv
[09:01:59]   scoring l1 / val_backward

=== val_backward  (prompt_len=1024, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   1/20    5.0%
     1.00   0/20    0.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the item with the special magic number 8056020.\n
  [MISS] expect=cactus
         got= the Zangoose.\n

  wrote lenb_len_1024/scores/l1_val_backward.csv
[09:01:59]   scoring l2 / niah_single

=== niah_single  (prompt_len=1024, n=100) ===
  accuracy: 96/100 = 96.0%
    depth      acc   (needle position in context)
     0.00  19/20   95.0%
     0.25  19/20   95.0%
     0.50  20/20  100.0%
     0.75  19/20   95.0%
     1.00  19/20   95.0%

  --- 2 sample generations ---
  [HIT ] expect=8056020
         got= 805602020.\n
  [HIT ] expect=5343902
         got= 5343902.  shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward

  wrote lenb_len_1024/scores/l2_niah_single.csv
[09:01:59]   scoring l2 / val_backward

=== val_backward  (prompt_len=1024, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   1/20    5.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   0/20    0.0%
     1.00   0/20    0.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the item with the special magic number 8056020.\n
  [MISS] expect=cactus
         got= the Zangoose.\n

  wrote lenb_len_1024/scores/l2_val_backward.csv
[09:01:59] building summary

=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
  task                 armA_c       l1       l2
  niah_single            94.0     95.0     96.0
  val_backward            1.0      1.0      1.0

  degenerate (<=1% for every model): ['val_backward']

=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
  depth             armA_c              l1              l2
  0.00          85.0+-15.4      90.0+-13.7      95.0+-11.4
  0.25         100.0+- 8.1     100.0+- 8.1      95.0+-11.4
  0.50          95.0+-11.4      95.0+-11.4     100.0+- 8.1
  0.75         100.0+- 8.1     100.0+- 8.1      95.0+-11.4
  1.00          90.0+-13.7      90.0+-13.7      95.0+-11.4

  wrote lenb_len_1024/summary.csv and lenb_len_1024/depth_summary.csv
[09:01:59] DONE -> lenb_len_1024/summary.csv