File size: 6,017 Bytes
b50ee6a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | [08:17:23] longeval: T=16384 prompt_len=1024 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[08:17:23] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin
[08:17:23] prompt signature changed -- regenerating prompts and clearing state
[08:17:23] generating prompts: niah_single
[08:17:28] generating prompts: val_backward
[08:17:28] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[08:17:28] 6 job(s) to run
[08:17:28] [gpu 0] armA_c / niah_single
[08:17:28] [gpu 1] armA_c / val_backward
[08:17:28] [gpu 2] l1 / niah_single
[08:17:28] [gpu 3] l1 / val_backward
[08:17:28] [gpu 0] l2 / niah_single
[08:17:28] [gpu 1] l2 / val_backward
[08:20:56] NO OUTPUT armA_c/niah_single -- see lenb_len_1024/gen/armA_c_niah_single.log
[08:20:56] NO OUTPUT armA_c/val_backward -- see lenb_len_1024/gen/armA_c_val_backward.log
[08:20:56] NO OUTPUT l1/niah_single -- see lenb_len_1024/gen/l1_niah_single.log
[08:20:56] NO OUTPUT l1/val_backward -- see lenb_len_1024/gen/l1_val_backward.log
[08:20:56] NO OUTPUT l2/niah_single -- see lenb_len_1024/gen/l2_niah_single.log
[08:20:56] NO OUTPUT l2/val_backward -- see lenb_len_1024/gen/l2_val_backward.log
[08:20:56] building summary
no scores yet
[08:20:56] DONE -> lenb_len_1024/summary.csv
[08:56:22] longeval: T=16384 prompt_len=1024 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[08:56:22] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin
[08:56:22] prompt signature changed -- regenerating prompts and clearing state
[08:56:22] generating prompts: niah_single
[08:56:22] generating prompts: val_backward
[08:56:23] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[08:56:23] 6 job(s) to run
[08:56:23] [gpu 0] armA_c / niah_single
[08:56:23] [gpu 1] armA_c / val_backward
[08:56:23] [gpu 2] l1 / niah_single
[08:56:23] [gpu 3] l1 / val_backward
[08:56:23] [gpu 0] l2 / niah_single
[08:56:23] [gpu 1] l2 / val_backward
[09:01:58] scoring armA_c / niah_single
=== niah_single (prompt_len=1024, n=100) ===
accuracy: 94/100 = 94.0%
depth acc (needle position in context)
0.00 17/20 85.0%
0.25 20/20 100.0%
0.50 19/20 95.0%
0.75 20/20 100.0%
1.00 18/20 90.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 8056020.\n
[HIT ] expect=5343902
got= 5343902. shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward
wrote lenb_len_1024/scores/armA_c_niah_single.csv
[09:01:58] scoring armA_c / val_backward
=== val_backward (prompt_len=1024, n=100) ===
accuracy: 1/100 = 1.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 1/20 5.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the one that has the special magic number 8056020.\n
[MISS] expect=cactus
got= the Zangoose.\n
wrote lenb_len_1024/scores/armA_c_val_backward.csv
[09:01:58] scoring l1 / niah_single
=== niah_single (prompt_len=1024, n=100) ===
accuracy: 95/100 = 95.0%
depth acc (needle position in context)
0.00 18/20 90.0%
0.25 20/20 100.0%
0.50 19/20 95.0%
0.75 20/20 100.0%
1.00 18/20 90.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 805602020.\n
[HIT ] expect=5343902
got= 5343902. shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward
wrote lenb_len_1024/scores/l1_niah_single.csv
[09:01:59] scoring l1 / val_backward
=== val_backward (prompt_len=1024, n=100) ===
accuracy: 1/100 = 1.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 1/20 5.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the item with the special magic number 8056020.\n
[MISS] expect=cactus
got= the Zangoose.\n
wrote lenb_len_1024/scores/l1_val_backward.csv
[09:01:59] scoring l2 / niah_single
=== niah_single (prompt_len=1024, n=100) ===
accuracy: 96/100 = 96.0%
depth acc (needle position in context)
0.00 19/20 95.0%
0.25 19/20 95.0%
0.50 20/20 100.0%
0.75 19/20 95.0%
1.00 19/20 95.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 805602020.\n
[HIT ] expect=5343902
got= 5343902. shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward
wrote lenb_len_1024/scores/l2_niah_single.csv
[09:01:59] scoring l2 / val_backward
=== val_backward (prompt_len=1024, n=100) ===
accuracy: 1/100 = 1.0%
depth acc (needle position in context)
0.00 1/20 5.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the item with the special magic number 8056020.\n
[MISS] expect=cactus
got= the Zangoose.\n
wrote lenb_len_1024/scores/l2_val_backward.csv
[09:01:59] building summary
=== per-task accuracy % (secondary/descriptive: ~10 per depth cell) ===
task armA_c l1 l2
niah_single 94.0 95.0 96.0
val_backward 1.0 1.0 1.0
degenerate (<=1% for every model): ['val_backward']
=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
depth armA_c l1 l2
0.00 85.0+-15.4 90.0+-13.7 95.0+-11.4
0.25 100.0+- 8.1 100.0+- 8.1 95.0+-11.4
0.50 95.0+-11.4 95.0+-11.4 100.0+- 8.1
0.75 100.0+- 8.1 100.0+- 8.1 95.0+-11.4
1.00 90.0+-13.7 90.0+-13.7 95.0+-11.4
wrote lenb_len_1024/summary.csv and lenb_len_1024/depth_summary.csv
[09:01:59] DONE -> lenb_len_1024/summary.csv
|