File size: 4,905 Bytes
3163896
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
[08:56:22] longeval: T=16384 prompt_len=1024 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[08:56:22] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin
[08:56:22] prompt signature changed -- regenerating prompts and clearing state
[08:56:22] generating prompts: niah_single
  niah_single      100 x 1024 tok -> lenb_len_1024/prompts/niah_single_prompts.bin
[08:56:22] generating prompts: val_backward
  val_backward     100 x 1024 tok -> lenb_len_1024/prompts/val_backward_prompts.bin
[08:56:23] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[08:56:23] 6 job(s) to run
[08:56:23]   [gpu 0] armA_c / niah_single
[08:56:23]   [gpu 1] armA_c / val_backward
[08:56:23]   [gpu 2] l1 / niah_single
[08:56:23]   [gpu 3] l1 / val_backward
[08:56:23]   [gpu 0] l2 / niah_single
[08:56:23]   [gpu 1] l2 / val_backward
[09:01:58]   scoring armA_c / niah_single

=== niah_single  (prompt_len=1024, n=100) ===
  accuracy: 94/100 = 94.0%
    depth      acc   (needle position in context)
     0.00  17/20   85.0%
     0.25  20/20  100.0%
     0.50  19/20   95.0%
     0.75  20/20  100.0%
     1.00  18/20   90.0%

  --- 2 sample generations ---
  [HIT ] expect=8056020
         got= 8056020.\n
  [HIT ] expect=5343902
         got= 5343902.  shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward

  wrote lenb_len_1024/scores/armA_c_niah_single.csv
[09:01:58]   scoring armA_c / val_backward

=== val_backward  (prompt_len=1024, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   1/20    5.0%
     1.00   0/20    0.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the one that has the special magic number 8056020.\n
  [MISS] expect=cactus
         got= the Zangoose.\n

  wrote lenb_len_1024/scores/armA_c_val_backward.csv
[09:01:58]   scoring l1 / niah_single

=== niah_single  (prompt_len=1024, n=100) ===
  accuracy: 95/100 = 95.0%
    depth      acc   (needle position in context)
     0.00  18/20   90.0%
     0.25  20/20  100.0%
     0.50  19/20   95.0%
     0.75  20/20  100.0%
     1.00  18/20   90.0%

  --- 2 sample generations ---
  [HIT ] expect=8056020
         got= 805602020.\n
  [HIT ] expect=5343902
         got= 5343902.  shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward

  wrote lenb_len_1024/scores/l1_niah_single.csv
[09:01:59]   scoring l1 / val_backward

=== val_backward  (prompt_len=1024, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   1/20    5.0%
     1.00   0/20    0.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the item with the special magic number 8056020.\n
  [MISS] expect=cactus
         got= the Zangoose.\n

  wrote lenb_len_1024/scores/l1_val_backward.csv
[09:01:59]   scoring l2 / niah_single

=== niah_single  (prompt_len=1024, n=100) ===
  accuracy: 96/100 = 96.0%
    depth      acc   (needle position in context)
     0.00  19/20   95.0%
     0.25  19/20   95.0%
     0.50  20/20  100.0%
     0.75  19/20   95.0%
     1.00  19/20   95.0%

  --- 2 sample generations ---
  [HIT ] expect=8056020
         got= 805602020.\n
  [HIT ] expect=5343902
         got= 5343902.  shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward

  wrote lenb_len_1024/scores/l2_niah_single.csv
[09:01:59]   scoring l2 / val_backward

=== val_backward  (prompt_len=1024, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   1/20    5.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   0/20    0.0%
     1.00   0/20    0.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the item with the special magic number 8056020.\n
  [MISS] expect=cactus
         got= the Zangoose.\n

  wrote lenb_len_1024/scores/l2_val_backward.csv
[09:01:59] building summary

=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
  task                 armA_c       l1       l2
  niah_single            94.0     95.0     96.0
  val_backward            1.0      1.0      1.0

  degenerate (<=1% for every model): ['val_backward']

=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
  depth             armA_c              l1              l2
  0.00          85.0+-15.4      90.0+-13.7      95.0+-11.4
  0.25         100.0+- 8.1     100.0+- 8.1      95.0+-11.4
  0.50          95.0+-11.4      95.0+-11.4     100.0+- 8.1
  0.75         100.0+- 8.1     100.0+- 8.1      95.0+-11.4
  1.00          90.0+-13.7      90.0+-13.7      95.0+-11.4

  wrote lenb_len_1024/summary.csv and lenb_len_1024/depth_summary.csv
[09:01:59] DONE -> lenb_len_1024/summary.csv