File size: 4,905 Bytes
3163896 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 | [08:56:22] longeval: T=16384 prompt_len=1024 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[08:56:22] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin
[08:56:22] prompt signature changed -- regenerating prompts and clearing state
[08:56:22] generating prompts: niah_single
niah_single 100 x 1024 tok -> lenb_len_1024/prompts/niah_single_prompts.bin
[08:56:22] generating prompts: val_backward
val_backward 100 x 1024 tok -> lenb_len_1024/prompts/val_backward_prompts.bin
[08:56:23] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[08:56:23] 6 job(s) to run
[08:56:23] [gpu 0] armA_c / niah_single
[08:56:23] [gpu 1] armA_c / val_backward
[08:56:23] [gpu 2] l1 / niah_single
[08:56:23] [gpu 3] l1 / val_backward
[08:56:23] [gpu 0] l2 / niah_single
[08:56:23] [gpu 1] l2 / val_backward
[09:01:58] scoring armA_c / niah_single
=== niah_single (prompt_len=1024, n=100) ===
accuracy: 94/100 = 94.0%
depth acc (needle position in context)
0.00 17/20 85.0%
0.25 20/20 100.0%
0.50 19/20 95.0%
0.75 20/20 100.0%
1.00 18/20 90.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 8056020.\n
[HIT ] expect=5343902
got= 5343902. shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward
wrote lenb_len_1024/scores/armA_c_niah_single.csv
[09:01:58] scoring armA_c / val_backward
=== val_backward (prompt_len=1024, n=100) ===
accuracy: 1/100 = 1.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 1/20 5.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the one that has the special magic number 8056020.\n
[MISS] expect=cactus
got= the Zangoose.\n
wrote lenb_len_1024/scores/armA_c_val_backward.csv
[09:01:58] scoring l1 / niah_single
=== niah_single (prompt_len=1024, n=100) ===
accuracy: 95/100 = 95.0%
depth acc (needle position in context)
0.00 18/20 90.0%
0.25 20/20 100.0%
0.50 19/20 95.0%
0.75 20/20 100.0%
1.00 18/20 90.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 805602020.\n
[HIT ] expect=5343902
got= 5343902. shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward
wrote lenb_len_1024/scores/l1_niah_single.csv
[09:01:59] scoring l1 / val_backward
=== val_backward (prompt_len=1024, n=100) ===
accuracy: 1/100 = 1.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 1/20 5.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the item with the special magic number 8056020.\n
[MISS] expect=cactus
got= the Zangoose.\n
wrote lenb_len_1024/scores/l1_val_backward.csv
[09:01:59] scoring l2 / niah_single
=== niah_single (prompt_len=1024, n=100) ===
accuracy: 96/100 = 96.0%
depth acc (needle position in context)
0.00 19/20 95.0%
0.25 19/20 95.0%
0.50 20/20 100.0%
0.75 19/20 95.0%
1.00 19/20 95.0%
--- 2 sample generations ---
[HIT ] expect=8056020
got= 805602020.\n
[HIT ] expect=5343902
got= 5343902. shaped, pink physician raised his hand to his mouth in a gesture of silence to the awkward
wrote lenb_len_1024/scores/l2_niah_single.csv
[09:01:59] scoring l2 / val_backward
=== val_backward (prompt_len=1024, n=100) ===
accuracy: 1/100 = 1.0%
depth acc (needle position in context)
0.00 1/20 5.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the item with the special magic number 8056020.\n
[MISS] expect=cactus
got= the Zangoose.\n
wrote lenb_len_1024/scores/l2_val_backward.csv
[09:01:59] building summary
=== per-task accuracy % (secondary/descriptive: ~10 per depth cell) ===
task armA_c l1 l2
niah_single 94.0 95.0 96.0
val_backward 1.0 1.0 1.0
degenerate (<=1% for every model): ['val_backward']
=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
depth armA_c l1 l2
0.00 85.0+-15.4 90.0+-13.7 95.0+-11.4
0.25 100.0+- 8.1 100.0+- 8.1 95.0+-11.4
0.50 95.0+-11.4 95.0+-11.4 100.0+- 8.1
0.75 100.0+- 8.1 100.0+- 8.1 95.0+-11.4
1.00 90.0+-13.7 90.0+-13.7 95.0+-11.4
wrote lenb_len_1024/summary.csv and lenb_len_1024/depth_summary.csv
[09:01:59] DONE -> lenb_len_1024/summary.csv
|