File size: 7,505 Bytes
feb6e1c 95af676 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 | [08:04:43] longeval: T=16384 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1
[08:04:43] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
[08:04:43] prompt signature changed -- regenerating prompts and clearing state
[08:04:43] generating prompts: niah_single
niah_single 100 x 8192 tok -> decide_len_8192/prompts/niah_single_prompts.bin
[08:04:43] fan-out: 2 GPU(s), NP=1 -> 4 concurrent job(s)
[08:04:43] 4 job(s) to run
[08:04:43] [gpu 0] armA_c / niah_single
[08:04:43] [gpu 1] l1 / niah_single
[08:04:43] [gpu 0] l2 / niah_single
[08:04:43] [gpu 1] l3 / niah_single
[08:13:00] scoring armA_c / niah_single
=== niah_single (prompt_len=8192, n=100) ===
accuracy: 53/100 = 53.0%
depth acc (needle position in context)
0.00 3/20 15.0%
0.25 9/20 45.0%
0.50 11/20 55.0%
0.75 11/20 55.0%
1.00 19/20 95.0%
--- 2 sample generations ---
[MISS] expect=8056020
got= the number of the day.\n
[HIT ] expect=5343902
got= 5343902.\n
wrote decide_len_8192/scores/armA_c_niah_single.csv
[08:13:01] scoring l1 / niah_single
=== niah_single (prompt_len=8192, n=100) ===
accuracy: 48/100 = 48.0%
depth acc (needle position in context)
0.00 1/20 5.0%
0.25 7/20 35.0%
0.50 8/20 40.0%
0.75 14/20 70.0%
1.00 18/20 90.0%
--- 2 sample generations ---
[MISS] expect=8056020
got= 7.\nQuestion: What is the special magic number for yarrow?\n
[HIT ] expect=5343902
got= 5343902.\n
wrote decide_len_8192/scores/l1_niah_single.csv
[08:13:01] scoring l2 / niah_single
=== niah_single (prompt_len=8192, n=100) ===
accuracy: 62/100 = 62.0%
depth acc (needle position in context)
0.00 11/20 55.0%
0.25 11/20 55.0%
0.50 10/20 50.0%
0.75 11/20 55.0%
1.00 19/20 95.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 805602020.\n
[HIT ] expect=5343902
got= 5343902.\n
wrote decide_len_8192/scores/l2_niah_single.csv
[08:13:01] scoring l3 / niah_single
=== niah_single (prompt_len=8192, n=100) ===
accuracy: 64/100 = 64.0%
depth acc (needle position in context)
0.00 12/20 60.0%
0.25 12/20 60.0%
0.50 11/20 55.0%
0.75 11/20 55.0%
1.00 18/20 90.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 805602020.\n
[HIT ] expect=5343902
got= 5343902.\n
wrote decide_len_8192/scores/l3_niah_single.csv
[08:13:01] building summary
=== per-task accuracy % (secondary/descriptive: ~10 per depth cell) ===
task armA_c l1 l2 l3
niah_single 53.0 48.0 62.0 64.0
=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
depth armA_c l1 l2 l3
0.00 15.0+-15.4 5.0+-11.4 55.0+-20.0 60.0+-19.7
0.25 45.0+-20.0 35.0+-19.3 55.0+-20.0 60.0+-19.7
0.50 55.0+-20.0 40.0+-19.7 50.0+-20.1 55.0+-20.0
0.75 55.0+-20.0 70.0+-18.7 55.0+-20.0 55.0+-20.0
1.00 95.0+-11.4 90.0+-13.7 95.0+-11.4 90.0+-13.7
wrote decide_len_8192/summary.csv and decide_len_8192/depth_summary.csv
[08:13:01] DONE -> decide_len_8192/summary.csv
[13:10:40] longeval: T=16384 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1
[13:10:40] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
[13:10:40] fan-out: 2 GPU(s), NP=1 -> 4 concurrent job(s)
[13:10:40] armA_c/niah_single already generated
[13:10:40] l1/niah_single already generated
[13:10:40] l2/niah_single already generated
[13:10:40] l3/niah_single already generated
[13:10:40] 1 job(s) to run
[13:10:40] [gpu 0] l3long / niah_single
[13:12:56] scoring l3long / niah_single
=== niah_single (prompt_len=8192, n=100) ===
accuracy: 55/100 = 55.0%
depth acc (needle position in context)
0.00 6/20 30.0%
0.25 12/20 60.0%
0.50 8/20 40.0%
0.75 10/20 50.0%
1.00 19/20 95.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 8056020.\n
[HIT ] expect=5343902
got= 5343902.\n
wrote decide_len_8192/scores/l3long_niah_single.csv
[13:12:56] building summary
=== per-task accuracy % (secondary/descriptive: ~10 per depth cell) ===
task armA_c l1 l2 l3 l3long
niah_single 53.0 48.0 62.0 64.0 55.0
=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
depth armA_c l1 l2 l3 l3long
0.00 15.0+-15.4 5.0+-11.4 55.0+-20.0 60.0+-19.7 30.0+-18.7
0.25 45.0+-20.0 35.0+-19.3 55.0+-20.0 60.0+-19.7 60.0+-19.7
0.50 55.0+-20.0 40.0+-19.7 50.0+-20.1 55.0+-20.0 40.0+-19.7
0.75 55.0+-20.0 70.0+-18.7 55.0+-20.0 55.0+-20.0 50.0+-20.1
1.00 95.0+-11.4 90.0+-13.7 95.0+-11.4 90.0+-13.7 95.0+-11.4
wrote decide_len_8192/summary.csv and decide_len_8192/depth_summary.csv
[13:12:56] DONE -> decide_len_8192/summary.csv
[16:42:35] longeval: T=16384 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1
[16:42:35] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
[16:42:35] fan-out: 2 GPU(s), NP=1 -> 4 concurrent job(s)
[16:42:35] armA_c/niah_single already generated
[16:42:35] l1/niah_single already generated
[16:42:35] l2/niah_single already generated
[16:42:35] l3/niah_single already generated
[16:42:35] l3long/niah_single already generated
[16:42:35] 1 job(s) to run
[16:42:35] [gpu 0] l3long2k / niah_single
[16:44:43] scoring l3long2k / niah_single
=== niah_single (prompt_len=8192, n=100) ===
accuracy: 54/100 = 54.0%
depth acc (needle position in context)
0.00 6/20 30.0%
0.25 12/20 60.0%
0.50 10/20 50.0%
0.75 9/20 45.0%
1.00 17/20 85.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 8056020.\n
[HIT ] expect=5343902
got= 5343902.\n
wrote decide_len_8192/scores/l3long2k_niah_single.csv
[16:44:43] building summary
=== per-task accuracy % (secondary/descriptive: ~10 per depth cell) ===
task armA_c l1 l2 l3 l3long l3long2k
niah_single 53.0 48.0 62.0 64.0 55.0 54.0
=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
depth armA_c l1 l2 l3 l3long l3long2k
0.00 15.0+-15.4 5.0+-11.4 55.0+-20.0 60.0+-19.7 30.0+-18.7 30.0+-18.7
0.25 45.0+-20.0 35.0+-19.3 55.0+-20.0 60.0+-19.7 60.0+-19.7 60.0+-19.7
0.50 55.0+-20.0 40.0+-19.7 50.0+-20.1 55.0+-20.0 40.0+-19.7 50.0+-20.1
0.75 55.0+-20.0 70.0+-18.7 55.0+-20.0 55.0+-20.0 50.0+-20.1 45.0+-20.0
1.00 95.0+-11.4 90.0+-13.7 95.0+-11.4 90.0+-13.7 95.0+-11.4 85.0+-15.4
wrote decide_len_8192/summary.csv and decide_len_8192/depth_summary.csv
[16:44:43] DONE -> decide_len_8192/summary.csv
|