ZachW commited on
Commit
702351b
·
verified ·
1 Parent(s): a2d1b30

Upload extrapolation artifact: 050302 RL OP13-20 step200 graph aggregate results

Browse files
050302/evals/rl050302_op13_20_step200_merged_hf/graph_results/results.json ADDED
@@ -0,0 +1,1780 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "files": {
3
+ "rl050302_op13_20_step200_merged_hf_ood_generations.jsonl.graph_results.jsonl": {
4
+ "per_template": {
5
+ "crazy_zootopia": {
6
+ "per_op_pass_at_k": {
7
+ "11": {
8
+ "pass@1": 0.34844192634560905,
9
+ "pass@2": 0.38243626062322944,
10
+ "pass@4": 0.40793201133144474,
11
+ "pass@8": 0.42492917847025496,
12
+ "pass@16": 0.43059490084985835,
13
+ "pass@32": 0.44192634560906513,
14
+ "pass@64": 0.45042492917847027,
15
+ "pass@128": 0.45325779036827196
16
+ },
17
+ "12": {
18
+ "pass@1": 0.4180790960451977,
19
+ "pass@2": 0.4604519774011299,
20
+ "pass@4": 0.4745762711864407,
21
+ "pass@8": 0.4830508474576271,
22
+ "pass@16": 0.5084745762711864,
23
+ "pass@32": 0.519774011299435,
24
+ "pass@64": 0.5254237288135594,
25
+ "pass@128": 0.5282485875706214
26
+ },
27
+ "13": {
28
+ "pass@1": 0.2413793103448276,
29
+ "pass@2": 0.3103448275862069,
30
+ "pass@4": 0.3667711598746082,
31
+ "pass@8": 0.3887147335423197,
32
+ "pass@16": 0.41379310344827586,
33
+ "pass@32": 0.43260188087774293,
34
+ "pass@64": 0.445141065830721,
35
+ "pass@128": 0.45768025078369906
36
+ },
37
+ "14": {
38
+ "pass@1": 0.22741433021806853,
39
+ "pass@2": 0.2866043613707165,
40
+ "pass@4": 0.3302180685358255,
41
+ "pass@8": 0.35202492211838005,
42
+ "pass@16": 0.3644859813084112,
43
+ "pass@32": 0.40809968847352024,
44
+ "pass@64": 0.4205607476635514,
45
+ "pass@128": 0.42990654205607476
46
+ },
47
+ "15": {
48
+ "pass@1": 0.12138728323699421,
49
+ "pass@2": 0.17630057803468208,
50
+ "pass@4": 0.21965317919075145,
51
+ "pass@8": 0.25722543352601157,
52
+ "pass@16": 0.2947976878612717,
53
+ "pass@32": 0.30057803468208094,
54
+ "pass@64": 0.3236994219653179,
55
+ "pass@128": 0.3439306358381503
56
+ },
57
+ "16": {
58
+ "pass@1": 0.11290322580645161,
59
+ "pass@2": 0.16451612903225807,
60
+ "pass@4": 0.2129032258064516,
61
+ "pass@8": 0.23870967741935484,
62
+ "pass@16": 0.25161290322580643,
63
+ "pass@32": 0.2806451612903226,
64
+ "pass@64": 0.3096774193548387,
65
+ "pass@128": 0.34516129032258064
66
+ },
67
+ "17": {
68
+ "pass@1": 0.046242774566473986,
69
+ "pass@2": 0.06936416184971098,
70
+ "pass@4": 0.09826589595375723,
71
+ "pass@8": 0.13583815028901733,
72
+ "pass@16": 0.17052023121387283,
73
+ "pass@32": 0.2023121387283237,
74
+ "pass@64": 0.23121387283236994,
75
+ "pass@128": 0.2543352601156069
76
+ },
77
+ "18": {
78
+ "pass@1": 0.025714285714285714,
79
+ "pass@2": 0.037142857142857144,
80
+ "pass@4": 0.04857142857142857,
81
+ "pass@8": 0.07428571428571429,
82
+ "pass@16": 0.09714285714285714,
83
+ "pass@32": 0.12285714285714286,
84
+ "pass@64": 0.13428571428571429,
85
+ "pass@128": 0.15714285714285714
86
+ },
87
+ "19": {
88
+ "pass@1": 0.009009009009009009,
89
+ "pass@2": 0.009009009009009009,
90
+ "pass@4": 0.015015015015015015,
91
+ "pass@8": 0.021021021021021023,
92
+ "pass@16": 0.02702702702702703,
93
+ "pass@32": 0.042042042042042045,
94
+ "pass@64": 0.05105105105105105,
95
+ "pass@128": 0.06906906906906907
96
+ },
97
+ "20": {
98
+ "pass@1": 0.0,
99
+ "pass@2": 0.012012012012012012,
100
+ "pass@4": 0.024024024024024024,
101
+ "pass@8": 0.03303303303303303,
102
+ "pass@16": 0.03303303303303303,
103
+ "pass@32": 0.042042042042042045,
104
+ "pass@64": 0.05105105105105105,
105
+ "pass@128": 0.06306306306306306
106
+ }
107
+ },
108
+ "per_op_pass_at_columns": {
109
+ "pass@1": {
110
+ "11": 0.34844192634560905,
111
+ "12": 0.4180790960451977,
112
+ "13": 0.2413793103448276,
113
+ "14": 0.22741433021806853,
114
+ "15": 0.12138728323699421,
115
+ "16": 0.11290322580645161,
116
+ "17": 0.046242774566473986,
117
+ "18": 0.025714285714285714,
118
+ "19": 0.009009009009009009,
119
+ "20": 0.0
120
+ },
121
+ "pass@2": {
122
+ "11": 0.38243626062322944,
123
+ "12": 0.4604519774011299,
124
+ "13": 0.3103448275862069,
125
+ "14": 0.2866043613707165,
126
+ "15": 0.17630057803468208,
127
+ "16": 0.16451612903225807,
128
+ "17": 0.06936416184971098,
129
+ "18": 0.037142857142857144,
130
+ "19": 0.009009009009009009,
131
+ "20": 0.012012012012012012
132
+ },
133
+ "pass@4": {
134
+ "11": 0.40793201133144474,
135
+ "12": 0.4745762711864407,
136
+ "13": 0.3667711598746082,
137
+ "14": 0.3302180685358255,
138
+ "15": 0.21965317919075145,
139
+ "16": 0.2129032258064516,
140
+ "17": 0.09826589595375723,
141
+ "18": 0.04857142857142857,
142
+ "19": 0.015015015015015015,
143
+ "20": 0.024024024024024024
144
+ },
145
+ "pass@8": {
146
+ "11": 0.42492917847025496,
147
+ "12": 0.4830508474576271,
148
+ "13": 0.3887147335423197,
149
+ "14": 0.35202492211838005,
150
+ "15": 0.25722543352601157,
151
+ "16": 0.23870967741935484,
152
+ "17": 0.13583815028901733,
153
+ "18": 0.07428571428571429,
154
+ "19": 0.021021021021021023,
155
+ "20": 0.03303303303303303
156
+ },
157
+ "pass@16": {
158
+ "11": 0.43059490084985835,
159
+ "12": 0.5084745762711864,
160
+ "13": 0.41379310344827586,
161
+ "14": 0.3644859813084112,
162
+ "15": 0.2947976878612717,
163
+ "16": 0.25161290322580643,
164
+ "17": 0.17052023121387283,
165
+ "18": 0.09714285714285714,
166
+ "19": 0.02702702702702703,
167
+ "20": 0.03303303303303303
168
+ },
169
+ "pass@32": {
170
+ "11": 0.44192634560906513,
171
+ "12": 0.519774011299435,
172
+ "13": 0.43260188087774293,
173
+ "14": 0.40809968847352024,
174
+ "15": 0.30057803468208094,
175
+ "16": 0.2806451612903226,
176
+ "17": 0.2023121387283237,
177
+ "18": 0.12285714285714286,
178
+ "19": 0.042042042042042045,
179
+ "20": 0.042042042042042045
180
+ },
181
+ "pass@64": {
182
+ "11": 0.45042492917847027,
183
+ "12": 0.5254237288135594,
184
+ "13": 0.445141065830721,
185
+ "14": 0.4205607476635514,
186
+ "15": 0.3236994219653179,
187
+ "16": 0.3096774193548387,
188
+ "17": 0.23121387283236994,
189
+ "18": 0.13428571428571429,
190
+ "19": 0.05105105105105105,
191
+ "20": 0.05105105105105105
192
+ },
193
+ "pass@128": {
194
+ "11": 0.45325779036827196,
195
+ "12": 0.5282485875706214,
196
+ "13": 0.45768025078369906,
197
+ "14": 0.42990654205607476,
198
+ "15": 0.3439306358381503,
199
+ "16": 0.34516129032258064,
200
+ "17": 0.2543352601156069,
201
+ "18": 0.15714285714285714,
202
+ "19": 0.06906906906906907,
203
+ "20": 0.06306306306306306
204
+ }
205
+ },
206
+ "total_pass_at_k": {
207
+ "pass@1": 0.1563150074294205,
208
+ "pass@2": 0.19167904903417535,
209
+ "pass@4": 0.2202080237741456,
210
+ "pass@8": 0.2413075780089153,
211
+ "pass@16": 0.2597325408618128,
212
+ "pass@32": 0.2796433878157504,
213
+ "pass@64": 0.29450222882615157,
214
+ "pass@128": 0.31025260029717683
215
+ }
216
+ },
217
+ "movie_festival_awards": {
218
+ "per_op_pass_at_k": {
219
+ "11": {
220
+ "pass@1": 0.37142857142857144,
221
+ "pass@2": 0.40634920634920635,
222
+ "pass@4": 0.43492063492063493,
223
+ "pass@8": 0.4507936507936508,
224
+ "pass@16": 0.4666666666666667,
225
+ "pass@32": 0.4857142857142857,
226
+ "pass@64": 0.49523809523809526,
227
+ "pass@128": 0.5111111111111111
228
+ },
229
+ "12": {
230
+ "pass@1": 0.40061162079510704,
231
+ "pass@2": 0.44954128440366975,
232
+ "pass@4": 0.4617737003058104,
233
+ "pass@8": 0.4831804281345566,
234
+ "pass@16": 0.5045871559633027,
235
+ "pass@32": 0.5168195718654435,
236
+ "pass@64": 0.5229357798165137,
237
+ "pass@128": 0.5259938837920489
238
+ },
239
+ "13": {
240
+ "pass@1": 0.29012345679012347,
241
+ "pass@2": 0.3271604938271605,
242
+ "pass@4": 0.36728395061728397,
243
+ "pass@8": 0.3950617283950617,
244
+ "pass@16": 0.41358024691358025,
245
+ "pass@32": 0.4228395061728395,
246
+ "pass@64": 0.4382716049382716,
247
+ "pass@128": 0.4567901234567901
248
+ },
249
+ "14": {
250
+ "pass@1": 0.17804154302670624,
251
+ "pass@2": 0.2314540059347181,
252
+ "pass@4": 0.2789317507418398,
253
+ "pass@8": 0.29376854599406527,
254
+ "pass@16": 0.3115727002967359,
255
+ "pass@32": 0.33827893175074186,
256
+ "pass@64": 0.3590504451038576,
257
+ "pass@128": 0.3827893175074184
258
+ },
259
+ "15": {
260
+ "pass@1": 0.14814814814814814,
261
+ "pass@2": 0.19135802469135801,
262
+ "pass@4": 0.21604938271604937,
263
+ "pass@8": 0.23765432098765432,
264
+ "pass@16": 0.26851851851851855,
265
+ "pass@32": 0.2932098765432099,
266
+ "pass@64": 0.3055555555555556,
267
+ "pass@128": 0.30864197530864196
268
+ },
269
+ "16": {
270
+ "pass@1": 0.10914454277286136,
271
+ "pass@2": 0.14454277286135694,
272
+ "pass@4": 0.1976401179941003,
273
+ "pass@8": 0.22418879056047197,
274
+ "pass@16": 0.2536873156342183,
275
+ "pass@32": 0.2890855457227139,
276
+ "pass@64": 0.32448377581120946,
277
+ "pass@128": 0.3480825958702065
278
+ },
279
+ "17": {
280
+ "pass@1": 0.03313253012048193,
281
+ "pass@2": 0.045180722891566265,
282
+ "pass@4": 0.08132530120481928,
283
+ "pass@8": 0.11144578313253012,
284
+ "pass@16": 0.13253012048192772,
285
+ "pass@32": 0.1566265060240964,
286
+ "pass@64": 0.18373493975903615,
287
+ "pass@128": 0.19879518072289157
288
+ },
289
+ "18": {
290
+ "pass@1": 0.03225806451612903,
291
+ "pass@2": 0.04516129032258064,
292
+ "pass@4": 0.06774193548387097,
293
+ "pass@8": 0.08709677419354839,
294
+ "pass@16": 0.0935483870967742,
295
+ "pass@32": 0.11290322580645161,
296
+ "pass@64": 0.12580645161290321,
297
+ "pass@128": 0.13225806451612904
298
+ },
299
+ "19": {
300
+ "pass@1": 0.0030211480362537764,
301
+ "pass@2": 0.012084592145015106,
302
+ "pass@4": 0.015105740181268883,
303
+ "pass@8": 0.021148036253776436,
304
+ "pass@16": 0.027190332326283987,
305
+ "pass@32": 0.03625377643504532,
306
+ "pass@64": 0.045317220543806644,
307
+ "pass@128": 0.0513595166163142
308
+ },
309
+ "20": {
310
+ "pass@1": 0.0,
311
+ "pass@2": 0.0,
312
+ "pass@4": 0.003003003003003003,
313
+ "pass@8": 0.003003003003003003,
314
+ "pass@16": 0.003003003003003003,
315
+ "pass@32": 0.009009009009009009,
316
+ "pass@64": 0.015015015015015015,
317
+ "pass@128": 0.024024024024024024
318
+ }
319
+ },
320
+ "per_op_pass_at_columns": {
321
+ "pass@1": {
322
+ "11": 0.37142857142857144,
323
+ "12": 0.40061162079510704,
324
+ "13": 0.29012345679012347,
325
+ "14": 0.17804154302670624,
326
+ "15": 0.14814814814814814,
327
+ "16": 0.10914454277286136,
328
+ "17": 0.03313253012048193,
329
+ "18": 0.03225806451612903,
330
+ "19": 0.0030211480362537764,
331
+ "20": 0.0
332
+ },
333
+ "pass@2": {
334
+ "11": 0.40634920634920635,
335
+ "12": 0.44954128440366975,
336
+ "13": 0.3271604938271605,
337
+ "14": 0.2314540059347181,
338
+ "15": 0.19135802469135801,
339
+ "16": 0.14454277286135694,
340
+ "17": 0.045180722891566265,
341
+ "18": 0.04516129032258064,
342
+ "19": 0.012084592145015106,
343
+ "20": 0.0
344
+ },
345
+ "pass@4": {
346
+ "11": 0.43492063492063493,
347
+ "12": 0.4617737003058104,
348
+ "13": 0.36728395061728397,
349
+ "14": 0.2789317507418398,
350
+ "15": 0.21604938271604937,
351
+ "16": 0.1976401179941003,
352
+ "17": 0.08132530120481928,
353
+ "18": 0.06774193548387097,
354
+ "19": 0.015105740181268883,
355
+ "20": 0.003003003003003003
356
+ },
357
+ "pass@8": {
358
+ "11": 0.4507936507936508,
359
+ "12": 0.4831804281345566,
360
+ "13": 0.3950617283950617,
361
+ "14": 0.29376854599406527,
362
+ "15": 0.23765432098765432,
363
+ "16": 0.22418879056047197,
364
+ "17": 0.11144578313253012,
365
+ "18": 0.08709677419354839,
366
+ "19": 0.021148036253776436,
367
+ "20": 0.003003003003003003
368
+ },
369
+ "pass@16": {
370
+ "11": 0.4666666666666667,
371
+ "12": 0.5045871559633027,
372
+ "13": 0.41358024691358025,
373
+ "14": 0.3115727002967359,
374
+ "15": 0.26851851851851855,
375
+ "16": 0.2536873156342183,
376
+ "17": 0.13253012048192772,
377
+ "18": 0.0935483870967742,
378
+ "19": 0.027190332326283987,
379
+ "20": 0.003003003003003003
380
+ },
381
+ "pass@32": {
382
+ "11": 0.4857142857142857,
383
+ "12": 0.5168195718654435,
384
+ "13": 0.4228395061728395,
385
+ "14": 0.33827893175074186,
386
+ "15": 0.2932098765432099,
387
+ "16": 0.2890855457227139,
388
+ "17": 0.1566265060240964,
389
+ "18": 0.11290322580645161,
390
+ "19": 0.03625377643504532,
391
+ "20": 0.009009009009009009
392
+ },
393
+ "pass@64": {
394
+ "11": 0.49523809523809526,
395
+ "12": 0.5229357798165137,
396
+ "13": 0.4382716049382716,
397
+ "14": 0.3590504451038576,
398
+ "15": 0.3055555555555556,
399
+ "16": 0.32448377581120946,
400
+ "17": 0.18373493975903615,
401
+ "18": 0.12580645161290321,
402
+ "19": 0.045317220543806644,
403
+ "20": 0.015015015015015015
404
+ },
405
+ "pass@128": {
406
+ "11": 0.5111111111111111,
407
+ "12": 0.5259938837920489,
408
+ "13": 0.4567901234567901,
409
+ "14": 0.3827893175074184,
410
+ "15": 0.30864197530864196,
411
+ "16": 0.3480825958702065,
412
+ "17": 0.19879518072289157,
413
+ "18": 0.13225806451612904,
414
+ "19": 0.0513595166163142,
415
+ "20": 0.024024024024024024
416
+ }
417
+ },
418
+ "total_pass_at_k": {
419
+ "pass@1": 0.15556234718826406,
420
+ "pass@2": 0.18429095354523228,
421
+ "pass@4": 0.2114914425427873,
422
+ "pass@8": 0.22982885085574573,
423
+ "pass@16": 0.24663814180929094,
424
+ "pass@32": 0.265281173594132,
425
+ "pass@64": 0.2808679706601467,
426
+ "pass@128": 0.293398533007335
427
+ }
428
+ },
429
+ "teachers_in_school": {
430
+ "per_op_pass_at_k": {
431
+ "11": {
432
+ "pass@1": 0.43373493975903615,
433
+ "pass@2": 0.4819277108433735,
434
+ "pass@4": 0.5,
435
+ "pass@8": 0.5180722891566265,
436
+ "pass@16": 0.5331325301204819,
437
+ "pass@32": 0.5542168674698795,
438
+ "pass@64": 0.5692771084337349,
439
+ "pass@128": 0.5692771084337349
440
+ },
441
+ "12": {
442
+ "pass@1": 0.3385579937304075,
443
+ "pass@2": 0.36990595611285265,
444
+ "pass@4": 0.3887147335423197,
445
+ "pass@8": 0.4200626959247649,
446
+ "pass@16": 0.4263322884012539,
447
+ "pass@32": 0.43573667711598746,
448
+ "pass@64": 0.45141065830721006,
449
+ "pass@128": 0.45768025078369906
450
+ },
451
+ "13": {
452
+ "pass@1": 0.30532212885154064,
453
+ "pass@2": 0.3641456582633053,
454
+ "pass@4": 0.37815126050420167,
455
+ "pass@8": 0.38935574229691877,
456
+ "pass@16": 0.41456582633053224,
457
+ "pass@32": 0.43137254901960786,
458
+ "pass@64": 0.4481792717086835,
459
+ "pass@128": 0.46218487394957986
460
+ },
461
+ "14": {
462
+ "pass@1": 0.21929824561403508,
463
+ "pass@2": 0.2631578947368421,
464
+ "pass@4": 0.30409356725146197,
465
+ "pass@8": 0.34502923976608185,
466
+ "pass@16": 0.3742690058479532,
467
+ "pass@32": 0.4064327485380117,
468
+ "pass@64": 0.4327485380116959,
469
+ "pass@128": 0.43859649122807015
470
+ },
471
+ "15": {
472
+ "pass@1": 0.13333333333333333,
473
+ "pass@2": 0.18787878787878787,
474
+ "pass@4": 0.23333333333333334,
475
+ "pass@8": 0.2727272727272727,
476
+ "pass@16": 0.30606060606060603,
477
+ "pass@32": 0.3242424242424242,
478
+ "pass@64": 0.3393939393939394,
479
+ "pass@128": 0.35454545454545455
480
+ },
481
+ "16": {
482
+ "pass@1": 0.10541310541310542,
483
+ "pass@2": 0.14245014245014245,
484
+ "pass@4": 0.18233618233618235,
485
+ "pass@8": 0.2336182336182336,
486
+ "pass@16": 0.2678062678062678,
487
+ "pass@32": 0.31054131054131057,
488
+ "pass@64": 0.33903133903133903,
489
+ "pass@128": 0.36182336182336183
490
+ },
491
+ "17": {
492
+ "pass@1": 0.06521739130434782,
493
+ "pass@2": 0.08385093167701864,
494
+ "pass@4": 0.11490683229813664,
495
+ "pass@8": 0.14906832298136646,
496
+ "pass@16": 0.17080745341614906,
497
+ "pass@32": 0.18944099378881987,
498
+ "pass@64": 0.21428571428571427,
499
+ "pass@128": 0.2267080745341615
500
+ },
501
+ "18": {
502
+ "pass@1": 0.047058823529411764,
503
+ "pass@2": 0.07058823529411765,
504
+ "pass@4": 0.08235294117647059,
505
+ "pass@8": 0.1,
506
+ "pass@16": 0.12352941176470589,
507
+ "pass@32": 0.13823529411764707,
508
+ "pass@64": 0.15588235294117647,
509
+ "pass@128": 0.17941176470588235
510
+ },
511
+ "19": {
512
+ "pass@1": 0.005952380952380952,
513
+ "pass@2": 0.01488095238095238,
514
+ "pass@4": 0.020833333333333332,
515
+ "pass@8": 0.03571428571428571,
516
+ "pass@16": 0.03869047619047619,
517
+ "pass@32": 0.047619047619047616,
518
+ "pass@64": 0.050595238095238096,
519
+ "pass@128": 0.06845238095238096
520
+ },
521
+ "20": {
522
+ "pass@1": 0.0029940119760479044,
523
+ "pass@2": 0.005988023952095809,
524
+ "pass@4": 0.005988023952095809,
525
+ "pass@8": 0.005988023952095809,
526
+ "pass@16": 0.014970059880239521,
527
+ "pass@32": 0.023952095808383235,
528
+ "pass@64": 0.029940119760479042,
529
+ "pass@128": 0.041916167664670656
530
+ }
531
+ },
532
+ "per_op_pass_at_columns": {
533
+ "pass@1": {
534
+ "11": 0.43373493975903615,
535
+ "12": 0.3385579937304075,
536
+ "13": 0.30532212885154064,
537
+ "14": 0.21929824561403508,
538
+ "15": 0.13333333333333333,
539
+ "16": 0.10541310541310542,
540
+ "17": 0.06521739130434782,
541
+ "18": 0.047058823529411764,
542
+ "19": 0.005952380952380952,
543
+ "20": 0.0029940119760479044
544
+ },
545
+ "pass@2": {
546
+ "11": 0.4819277108433735,
547
+ "12": 0.36990595611285265,
548
+ "13": 0.3641456582633053,
549
+ "14": 0.2631578947368421,
550
+ "15": 0.18787878787878787,
551
+ "16": 0.14245014245014245,
552
+ "17": 0.08385093167701864,
553
+ "18": 0.07058823529411765,
554
+ "19": 0.01488095238095238,
555
+ "20": 0.005988023952095809
556
+ },
557
+ "pass@4": {
558
+ "11": 0.5,
559
+ "12": 0.3887147335423197,
560
+ "13": 0.37815126050420167,
561
+ "14": 0.30409356725146197,
562
+ "15": 0.23333333333333334,
563
+ "16": 0.18233618233618235,
564
+ "17": 0.11490683229813664,
565
+ "18": 0.08235294117647059,
566
+ "19": 0.020833333333333332,
567
+ "20": 0.005988023952095809
568
+ },
569
+ "pass@8": {
570
+ "11": 0.5180722891566265,
571
+ "12": 0.4200626959247649,
572
+ "13": 0.38935574229691877,
573
+ "14": 0.34502923976608185,
574
+ "15": 0.2727272727272727,
575
+ "16": 0.2336182336182336,
576
+ "17": 0.14906832298136646,
577
+ "18": 0.1,
578
+ "19": 0.03571428571428571,
579
+ "20": 0.005988023952095809
580
+ },
581
+ "pass@16": {
582
+ "11": 0.5331325301204819,
583
+ "12": 0.4263322884012539,
584
+ "13": 0.41456582633053224,
585
+ "14": 0.3742690058479532,
586
+ "15": 0.30606060606060603,
587
+ "16": 0.2678062678062678,
588
+ "17": 0.17080745341614906,
589
+ "18": 0.12352941176470589,
590
+ "19": 0.03869047619047619,
591
+ "20": 0.014970059880239521
592
+ },
593
+ "pass@32": {
594
+ "11": 0.5542168674698795,
595
+ "12": 0.43573667711598746,
596
+ "13": 0.43137254901960786,
597
+ "14": 0.4064327485380117,
598
+ "15": 0.3242424242424242,
599
+ "16": 0.31054131054131057,
600
+ "17": 0.18944099378881987,
601
+ "18": 0.13823529411764707,
602
+ "19": 0.047619047619047616,
603
+ "20": 0.023952095808383235
604
+ },
605
+ "pass@64": {
606
+ "11": 0.5692771084337349,
607
+ "12": 0.45141065830721006,
608
+ "13": 0.4481792717086835,
609
+ "14": 0.4327485380116959,
610
+ "15": 0.3393939393939394,
611
+ "16": 0.33903133903133903,
612
+ "17": 0.21428571428571427,
613
+ "18": 0.15588235294117647,
614
+ "19": 0.050595238095238096,
615
+ "20": 0.029940119760479042
616
+ },
617
+ "pass@128": {
618
+ "11": 0.5692771084337349,
619
+ "12": 0.45768025078369906,
620
+ "13": 0.46218487394957986,
621
+ "14": 0.43859649122807015,
622
+ "15": 0.35454545454545455,
623
+ "16": 0.36182336182336183,
624
+ "17": 0.2267080745341615,
625
+ "18": 0.17941176470588235,
626
+ "19": 0.06845238095238096,
627
+ "20": 0.041916167664670656
628
+ }
629
+ },
630
+ "total_pass_at_k": {
631
+ "pass@1": 0.1656259292298543,
632
+ "pass@2": 0.19863217365447516,
633
+ "pass@4": 0.22123104371097235,
634
+ "pass@8": 0.24710080285459413,
635
+ "pass@16": 0.2673208444840916,
636
+ "pass@32": 0.28664882545346415,
637
+ "pass@64": 0.30359797799583704,
638
+ "pass@128": 0.31668153434433544
639
+ }
640
+ }
641
+ },
642
+ "total": {
643
+ "per_op_pass_at_k": {
644
+ "11": {
645
+ "pass@1": 0.384,
646
+ "pass@2": 0.423,
647
+ "pass@4": 0.447,
648
+ "pass@8": 0.464,
649
+ "pass@16": 0.476,
650
+ "pass@32": 0.493,
651
+ "pass@64": 0.504,
652
+ "pass@128": 0.51
653
+ },
654
+ "12": {
655
+ "pass@1": 0.387,
656
+ "pass@2": 0.428,
657
+ "pass@4": 0.443,
658
+ "pass@8": 0.463,
659
+ "pass@16": 0.481,
660
+ "pass@32": 0.492,
661
+ "pass@64": 0.501,
662
+ "pass@128": 0.505
663
+ },
664
+ "13": {
665
+ "pass@1": 0.28,
666
+ "pass@2": 0.335,
667
+ "pass@4": 0.371,
668
+ "pass@8": 0.391,
669
+ "pass@16": 0.414,
670
+ "pass@32": 0.429,
671
+ "pass@64": 0.444,
672
+ "pass@128": 0.459
673
+ },
674
+ "14": {
675
+ "pass@1": 0.208,
676
+ "pass@2": 0.26,
677
+ "pass@4": 0.304,
678
+ "pass@8": 0.33,
679
+ "pass@16": 0.35,
680
+ "pass@32": 0.384,
681
+ "pass@64": 0.404,
682
+ "pass@128": 0.417
683
+ },
684
+ "15": {
685
+ "pass@1": 0.134,
686
+ "pass@2": 0.185,
687
+ "pass@4": 0.223,
688
+ "pass@8": 0.256,
689
+ "pass@16": 0.29,
690
+ "pass@32": 0.306,
691
+ "pass@64": 0.323,
692
+ "pass@128": 0.336
693
+ },
694
+ "16": {
695
+ "pass@1": 0.109,
696
+ "pass@2": 0.15,
697
+ "pass@4": 0.197,
698
+ "pass@8": 0.232,
699
+ "pass@16": 0.258,
700
+ "pass@32": 0.294,
701
+ "pass@64": 0.325,
702
+ "pass@128": 0.352
703
+ },
704
+ "17": {
705
+ "pass@1": 0.048,
706
+ "pass@2": 0.066,
707
+ "pass@4": 0.098,
708
+ "pass@8": 0.132,
709
+ "pass@16": 0.158,
710
+ "pass@32": 0.183,
711
+ "pass@64": 0.21,
712
+ "pass@128": 0.227
713
+ },
714
+ "18": {
715
+ "pass@1": 0.035,
716
+ "pass@2": 0.051,
717
+ "pass@4": 0.066,
718
+ "pass@8": 0.087,
719
+ "pass@16": 0.105,
720
+ "pass@32": 0.125,
721
+ "pass@64": 0.139,
722
+ "pass@128": 0.157
723
+ },
724
+ "19": {
725
+ "pass@1": 0.006,
726
+ "pass@2": 0.012,
727
+ "pass@4": 0.017,
728
+ "pass@8": 0.026,
729
+ "pass@16": 0.031,
730
+ "pass@32": 0.042,
731
+ "pass@64": 0.049,
732
+ "pass@128": 0.063
733
+ },
734
+ "20": {
735
+ "pass@1": 0.001,
736
+ "pass@2": 0.006,
737
+ "pass@4": 0.011,
738
+ "pass@8": 0.014,
739
+ "pass@16": 0.017,
740
+ "pass@32": 0.025,
741
+ "pass@64": 0.032,
742
+ "pass@128": 0.043
743
+ }
744
+ },
745
+ "per_op_pass_at_columns": {
746
+ "pass@1": {
747
+ "11": 0.384,
748
+ "12": 0.387,
749
+ "13": 0.28,
750
+ "14": 0.208,
751
+ "15": 0.134,
752
+ "16": 0.109,
753
+ "17": 0.048,
754
+ "18": 0.035,
755
+ "19": 0.006,
756
+ "20": 0.001
757
+ },
758
+ "pass@2": {
759
+ "11": 0.423,
760
+ "12": 0.428,
761
+ "13": 0.335,
762
+ "14": 0.26,
763
+ "15": 0.185,
764
+ "16": 0.15,
765
+ "17": 0.066,
766
+ "18": 0.051,
767
+ "19": 0.012,
768
+ "20": 0.006
769
+ },
770
+ "pass@4": {
771
+ "11": 0.447,
772
+ "12": 0.443,
773
+ "13": 0.371,
774
+ "14": 0.304,
775
+ "15": 0.223,
776
+ "16": 0.197,
777
+ "17": 0.098,
778
+ "18": 0.066,
779
+ "19": 0.017,
780
+ "20": 0.011
781
+ },
782
+ "pass@8": {
783
+ "11": 0.464,
784
+ "12": 0.463,
785
+ "13": 0.391,
786
+ "14": 0.33,
787
+ "15": 0.256,
788
+ "16": 0.232,
789
+ "17": 0.132,
790
+ "18": 0.087,
791
+ "19": 0.026,
792
+ "20": 0.014
793
+ },
794
+ "pass@16": {
795
+ "11": 0.476,
796
+ "12": 0.481,
797
+ "13": 0.414,
798
+ "14": 0.35,
799
+ "15": 0.29,
800
+ "16": 0.258,
801
+ "17": 0.158,
802
+ "18": 0.105,
803
+ "19": 0.031,
804
+ "20": 0.017
805
+ },
806
+ "pass@32": {
807
+ "11": 0.493,
808
+ "12": 0.492,
809
+ "13": 0.429,
810
+ "14": 0.384,
811
+ "15": 0.306,
812
+ "16": 0.294,
813
+ "17": 0.183,
814
+ "18": 0.125,
815
+ "19": 0.042,
816
+ "20": 0.025
817
+ },
818
+ "pass@64": {
819
+ "11": 0.504,
820
+ "12": 0.501,
821
+ "13": 0.444,
822
+ "14": 0.404,
823
+ "15": 0.323,
824
+ "16": 0.325,
825
+ "17": 0.21,
826
+ "18": 0.139,
827
+ "19": 0.049,
828
+ "20": 0.032
829
+ },
830
+ "pass@128": {
831
+ "11": 0.51,
832
+ "12": 0.505,
833
+ "13": 0.459,
834
+ "14": 0.417,
835
+ "15": 0.336,
836
+ "16": 0.352,
837
+ "17": 0.227,
838
+ "18": 0.157,
839
+ "19": 0.063,
840
+ "20": 0.043
841
+ }
842
+ },
843
+ "total_pass_at_k": {
844
+ "pass@1": 0.1592,
845
+ "pass@2": 0.1916,
846
+ "pass@4": 0.2177,
847
+ "pass@8": 0.2395,
848
+ "pass@16": 0.258,
849
+ "pass@32": 0.2773,
850
+ "pass@64": 0.2931,
851
+ "pass@128": 0.3069
852
+ }
853
+ },
854
+ "ks": [
855
+ 1,
856
+ 2,
857
+ 4,
858
+ 8,
859
+ 16,
860
+ 32,
861
+ 64,
862
+ 128
863
+ ],
864
+ "prompt_counts": {
865
+ "per_template": {
866
+ "crazy_zootopia": {
867
+ "per_op": {
868
+ "11": 353,
869
+ "12": 354,
870
+ "13": 319,
871
+ "14": 321,
872
+ "15": 346,
873
+ "16": 310,
874
+ "17": 346,
875
+ "18": 350,
876
+ "19": 333,
877
+ "20": 333
878
+ },
879
+ "total": 3365
880
+ },
881
+ "movie_festival_awards": {
882
+ "per_op": {
883
+ "11": 315,
884
+ "12": 327,
885
+ "13": 324,
886
+ "14": 337,
887
+ "15": 324,
888
+ "16": 339,
889
+ "17": 332,
890
+ "18": 310,
891
+ "19": 331,
892
+ "20": 333
893
+ },
894
+ "total": 3272
895
+ },
896
+ "teachers_in_school": {
897
+ "per_op": {
898
+ "11": 332,
899
+ "12": 319,
900
+ "13": 357,
901
+ "14": 342,
902
+ "15": 330,
903
+ "16": 351,
904
+ "17": 322,
905
+ "18": 340,
906
+ "19": 336,
907
+ "20": 334
908
+ },
909
+ "total": 3363
910
+ }
911
+ },
912
+ "total": {
913
+ "per_op": {
914
+ "11": 1000,
915
+ "12": 1000,
916
+ "13": 1000,
917
+ "14": 1000,
918
+ "15": 1000,
919
+ "16": 1000,
920
+ "17": 1000,
921
+ "18": 1000,
922
+ "19": 1000,
923
+ "20": 1000
924
+ },
925
+ "total": 10000
926
+ }
927
+ }
928
+ },
929
+ "rl050302_op13_20_step200_merged_hf_id_generations.jsonl.graph_results.jsonl": {
930
+ "per_template": {
931
+ "crazy_zootopia": {
932
+ "per_op_pass_at_k": {
933
+ "2": {
934
+ "pass@1": 0.9158878504672897,
935
+ "pass@2": 0.9345794392523364,
936
+ "pass@4": 0.956386292834891,
937
+ "pass@8": 0.9657320872274143,
938
+ "pass@16": 0.9813084112149533,
939
+ "pass@32": 0.9844236760124611,
940
+ "pass@64": 0.9844236760124611,
941
+ "pass@128": 0.9875389408099688
942
+ },
943
+ "3": {
944
+ "pass@1": 0.8623853211009175,
945
+ "pass@2": 0.8837920489296636,
946
+ "pass@4": 0.908256880733945,
947
+ "pass@8": 0.9235474006116208,
948
+ "pass@16": 0.9327217125382263,
949
+ "pass@32": 0.9480122324159022,
950
+ "pass@64": 0.963302752293578,
951
+ "pass@128": 0.9724770642201835
952
+ },
953
+ "4": {
954
+ "pass@1": 0.678125,
955
+ "pass@2": 0.696875,
956
+ "pass@4": 0.7125,
957
+ "pass@8": 0.725,
958
+ "pass@16": 0.740625,
959
+ "pass@32": 0.7625,
960
+ "pass@64": 0.771875,
961
+ "pass@128": 0.778125
962
+ },
963
+ "5": {
964
+ "pass@1": 0.546031746031746,
965
+ "pass@2": 0.5650793650793651,
966
+ "pass@4": 0.5873015873015873,
967
+ "pass@8": 0.6063492063492063,
968
+ "pass@16": 0.6412698412698413,
969
+ "pass@32": 0.6698412698412698,
970
+ "pass@64": 0.692063492063492,
971
+ "pass@128": 0.7047619047619048
972
+ },
973
+ "6": {
974
+ "pass@1": 0.4820359281437126,
975
+ "pass@2": 0.5239520958083832,
976
+ "pass@4": 0.5508982035928144,
977
+ "pass@8": 0.5748502994011976,
978
+ "pass@16": 0.6077844311377245,
979
+ "pass@32": 0.6197604790419161,
980
+ "pass@64": 0.6347305389221557,
981
+ "pass@128": 0.6676646706586826
982
+ },
983
+ "7": {
984
+ "pass@1": 0.4822485207100592,
985
+ "pass@2": 0.5236686390532544,
986
+ "pass@4": 0.5473372781065089,
987
+ "pass@8": 0.5828402366863905,
988
+ "pass@16": 0.6094674556213018,
989
+ "pass@32": 0.6183431952662722,
990
+ "pass@64": 0.6242603550295858,
991
+ "pass@128": 0.6331360946745562
992
+ },
993
+ "8": {
994
+ "pass@1": 0.4519230769230769,
995
+ "pass@2": 0.46794871794871795,
996
+ "pass@4": 0.48717948717948717,
997
+ "pass@8": 0.4967948717948718,
998
+ "pass@16": 0.5032051282051282,
999
+ "pass@32": 0.5384615384615384,
1000
+ "pass@64": 0.5512820512820513,
1001
+ "pass@128": 0.5608974358974359
1002
+ },
1003
+ "9": {
1004
+ "pass@1": 0.4422535211267606,
1005
+ "pass@2": 0.4704225352112676,
1006
+ "pass@4": 0.5098591549295775,
1007
+ "pass@8": 0.5211267605633803,
1008
+ "pass@16": 0.5408450704225352,
1009
+ "pass@32": 0.5577464788732395,
1010
+ "pass@64": 0.5774647887323944,
1011
+ "pass@128": 0.6056338028169014
1012
+ },
1013
+ "10": {
1014
+ "pass@1": 0.4690265486725664,
1015
+ "pass@2": 0.4837758112094395,
1016
+ "pass@4": 0.5014749262536873,
1017
+ "pass@8": 0.5162241887905604,
1018
+ "pass@16": 0.5191740412979351,
1019
+ "pass@32": 0.5221238938053098,
1020
+ "pass@64": 0.5339233038348082,
1021
+ "pass@128": 0.5398230088495575
1022
+ }
1023
+ },
1024
+ "per_op_pass_at_columns": {
1025
+ "pass@1": {
1026
+ "2": 0.9158878504672897,
1027
+ "3": 0.8623853211009175,
1028
+ "4": 0.678125,
1029
+ "5": 0.546031746031746,
1030
+ "6": 0.4820359281437126,
1031
+ "7": 0.4822485207100592,
1032
+ "8": 0.4519230769230769,
1033
+ "9": 0.4422535211267606,
1034
+ "10": 0.4690265486725664
1035
+ },
1036
+ "pass@2": {
1037
+ "2": 0.9345794392523364,
1038
+ "3": 0.8837920489296636,
1039
+ "4": 0.696875,
1040
+ "5": 0.5650793650793651,
1041
+ "6": 0.5239520958083832,
1042
+ "7": 0.5236686390532544,
1043
+ "8": 0.46794871794871795,
1044
+ "9": 0.4704225352112676,
1045
+ "10": 0.4837758112094395
1046
+ },
1047
+ "pass@4": {
1048
+ "2": 0.956386292834891,
1049
+ "3": 0.908256880733945,
1050
+ "4": 0.7125,
1051
+ "5": 0.5873015873015873,
1052
+ "6": 0.5508982035928144,
1053
+ "7": 0.5473372781065089,
1054
+ "8": 0.48717948717948717,
1055
+ "9": 0.5098591549295775,
1056
+ "10": 0.5014749262536873
1057
+ },
1058
+ "pass@8": {
1059
+ "2": 0.9657320872274143,
1060
+ "3": 0.9235474006116208,
1061
+ "4": 0.725,
1062
+ "5": 0.6063492063492063,
1063
+ "6": 0.5748502994011976,
1064
+ "7": 0.5828402366863905,
1065
+ "8": 0.4967948717948718,
1066
+ "9": 0.5211267605633803,
1067
+ "10": 0.5162241887905604
1068
+ },
1069
+ "pass@16": {
1070
+ "2": 0.9813084112149533,
1071
+ "3": 0.9327217125382263,
1072
+ "4": 0.740625,
1073
+ "5": 0.6412698412698413,
1074
+ "6": 0.6077844311377245,
1075
+ "7": 0.6094674556213018,
1076
+ "8": 0.5032051282051282,
1077
+ "9": 0.5408450704225352,
1078
+ "10": 0.5191740412979351
1079
+ },
1080
+ "pass@32": {
1081
+ "2": 0.9844236760124611,
1082
+ "3": 0.9480122324159022,
1083
+ "4": 0.7625,
1084
+ "5": 0.6698412698412698,
1085
+ "6": 0.6197604790419161,
1086
+ "7": 0.6183431952662722,
1087
+ "8": 0.5384615384615384,
1088
+ "9": 0.5577464788732395,
1089
+ "10": 0.5221238938053098
1090
+ },
1091
+ "pass@64": {
1092
+ "2": 0.9844236760124611,
1093
+ "3": 0.963302752293578,
1094
+ "4": 0.771875,
1095
+ "5": 0.692063492063492,
1096
+ "6": 0.6347305389221557,
1097
+ "7": 0.6242603550295858,
1098
+ "8": 0.5512820512820513,
1099
+ "9": 0.5774647887323944,
1100
+ "10": 0.5339233038348082
1101
+ },
1102
+ "pass@128": {
1103
+ "2": 0.9875389408099688,
1104
+ "3": 0.9724770642201835,
1105
+ "4": 0.778125,
1106
+ "5": 0.7047619047619048,
1107
+ "6": 0.6676646706586826,
1108
+ "7": 0.6331360946745562,
1109
+ "8": 0.5608974358974359,
1110
+ "9": 0.6056338028169014,
1111
+ "10": 0.5398230088495575
1112
+ }
1113
+ },
1114
+ "total_pass_at_k": {
1115
+ "pass@1": 0.5896656534954408,
1116
+ "pass@2": 0.6143194866599122,
1117
+ "pass@4": 0.6379601485984465,
1118
+ "pass@8": 0.6548463356973995,
1119
+ "pass@16": 0.6730834177642688,
1120
+ "pass@32": 0.6889564336372846,
1121
+ "pass@64": 0.70145221209051,
1122
+ "pass@128": 0.7146234380276933
1123
+ }
1124
+ },
1125
+ "movie_festival_awards": {
1126
+ "per_op_pass_at_k": {
1127
+ "2": {
1128
+ "pass@1": 0.9028571428571428,
1129
+ "pass@2": 0.9257142857142857,
1130
+ "pass@4": 0.9485714285714286,
1131
+ "pass@8": 0.9542857142857143,
1132
+ "pass@16": 0.9685714285714285,
1133
+ "pass@32": 0.9742857142857143,
1134
+ "pass@64": 0.98,
1135
+ "pass@128": 0.9857142857142858
1136
+ },
1137
+ "3": {
1138
+ "pass@1": 0.8688046647230321,
1139
+ "pass@2": 0.8979591836734694,
1140
+ "pass@4": 0.9125364431486881,
1141
+ "pass@8": 0.9387755102040817,
1142
+ "pass@16": 0.9416909620991254,
1143
+ "pass@32": 0.9475218658892128,
1144
+ "pass@64": 0.956268221574344,
1145
+ "pass@128": 0.9650145772594753
1146
+ },
1147
+ "4": {
1148
+ "pass@1": 0.6666666666666666,
1149
+ "pass@2": 0.7076023391812866,
1150
+ "pass@4": 0.7339181286549707,
1151
+ "pass@8": 0.7543859649122807,
1152
+ "pass@16": 0.7690058479532164,
1153
+ "pass@32": 0.7865497076023392,
1154
+ "pass@64": 0.804093567251462,
1155
+ "pass@128": 0.8128654970760234
1156
+ },
1157
+ "5": {
1158
+ "pass@1": 0.5176470588235295,
1159
+ "pass@2": 0.5617647058823529,
1160
+ "pass@4": 0.5911764705882353,
1161
+ "pass@8": 0.611764705882353,
1162
+ "pass@16": 0.6352941176470588,
1163
+ "pass@32": 0.6529411764705882,
1164
+ "pass@64": 0.6705882352941176,
1165
+ "pass@128": 0.6882352941176471
1166
+ },
1167
+ "6": {
1168
+ "pass@1": 0.4889589905362776,
1169
+ "pass@2": 0.5205047318611987,
1170
+ "pass@4": 0.5678233438485805,
1171
+ "pass@8": 0.5993690851735016,
1172
+ "pass@16": 0.6182965299684543,
1173
+ "pass@32": 0.6277602523659306,
1174
+ "pass@64": 0.6466876971608833,
1175
+ "pass@128": 0.6561514195583596
1176
+ },
1177
+ "7": {
1178
+ "pass@1": 0.43037974683544306,
1179
+ "pass@2": 0.4525316455696203,
1180
+ "pass@4": 0.47468354430379744,
1181
+ "pass@8": 0.5031645569620253,
1182
+ "pass@16": 0.5253164556962026,
1183
+ "pass@32": 0.5474683544303798,
1184
+ "pass@64": 0.5632911392405063,
1185
+ "pass@128": 0.5791139240506329
1186
+ },
1187
+ "8": {
1188
+ "pass@1": 0.4114114114114114,
1189
+ "pass@2": 0.44144144144144143,
1190
+ "pass@4": 0.46546546546546547,
1191
+ "pass@8": 0.4894894894894895,
1192
+ "pass@16": 0.5045045045045045,
1193
+ "pass@32": 0.5375375375375375,
1194
+ "pass@64": 0.5525525525525525,
1195
+ "pass@128": 0.5615615615615616
1196
+ },
1197
+ "9": {
1198
+ "pass@1": 0.42765273311897106,
1199
+ "pass@2": 0.4694533762057878,
1200
+ "pass@4": 0.5080385852090032,
1201
+ "pass@8": 0.5273311897106109,
1202
+ "pass@16": 0.5434083601286174,
1203
+ "pass@32": 0.5562700964630225,
1204
+ "pass@64": 0.5659163987138264,
1205
+ "pass@128": 0.5755627009646302
1206
+ },
1207
+ "10": {
1208
+ "pass@1": 0.417910447761194,
1209
+ "pass@2": 0.4417910447761194,
1210
+ "pass@4": 0.4597014925373134,
1211
+ "pass@8": 0.4746268656716418,
1212
+ "pass@16": 0.4835820895522388,
1213
+ "pass@32": 0.49850746268656715,
1214
+ "pass@64": 0.49850746268656715,
1215
+ "pass@128": 0.5104477611940299
1216
+ }
1217
+ },
1218
+ "per_op_pass_at_columns": {
1219
+ "pass@1": {
1220
+ "2": 0.9028571428571428,
1221
+ "3": 0.8688046647230321,
1222
+ "4": 0.6666666666666666,
1223
+ "5": 0.5176470588235295,
1224
+ "6": 0.4889589905362776,
1225
+ "7": 0.43037974683544306,
1226
+ "8": 0.4114114114114114,
1227
+ "9": 0.42765273311897106,
1228
+ "10": 0.417910447761194
1229
+ },
1230
+ "pass@2": {
1231
+ "2": 0.9257142857142857,
1232
+ "3": 0.8979591836734694,
1233
+ "4": 0.7076023391812866,
1234
+ "5": 0.5617647058823529,
1235
+ "6": 0.5205047318611987,
1236
+ "7": 0.4525316455696203,
1237
+ "8": 0.44144144144144143,
1238
+ "9": 0.4694533762057878,
1239
+ "10": 0.4417910447761194
1240
+ },
1241
+ "pass@4": {
1242
+ "2": 0.9485714285714286,
1243
+ "3": 0.9125364431486881,
1244
+ "4": 0.7339181286549707,
1245
+ "5": 0.5911764705882353,
1246
+ "6": 0.5678233438485805,
1247
+ "7": 0.47468354430379744,
1248
+ "8": 0.46546546546546547,
1249
+ "9": 0.5080385852090032,
1250
+ "10": 0.4597014925373134
1251
+ },
1252
+ "pass@8": {
1253
+ "2": 0.9542857142857143,
1254
+ "3": 0.9387755102040817,
1255
+ "4": 0.7543859649122807,
1256
+ "5": 0.611764705882353,
1257
+ "6": 0.5993690851735016,
1258
+ "7": 0.5031645569620253,
1259
+ "8": 0.4894894894894895,
1260
+ "9": 0.5273311897106109,
1261
+ "10": 0.4746268656716418
1262
+ },
1263
+ "pass@16": {
1264
+ "2": 0.9685714285714285,
1265
+ "3": 0.9416909620991254,
1266
+ "4": 0.7690058479532164,
1267
+ "5": 0.6352941176470588,
1268
+ "6": 0.6182965299684543,
1269
+ "7": 0.5253164556962026,
1270
+ "8": 0.5045045045045045,
1271
+ "9": 0.5434083601286174,
1272
+ "10": 0.4835820895522388
1273
+ },
1274
+ "pass@32": {
1275
+ "2": 0.9742857142857143,
1276
+ "3": 0.9475218658892128,
1277
+ "4": 0.7865497076023392,
1278
+ "5": 0.6529411764705882,
1279
+ "6": 0.6277602523659306,
1280
+ "7": 0.5474683544303798,
1281
+ "8": 0.5375375375375375,
1282
+ "9": 0.5562700964630225,
1283
+ "10": 0.49850746268656715
1284
+ },
1285
+ "pass@64": {
1286
+ "2": 0.98,
1287
+ "3": 0.956268221574344,
1288
+ "4": 0.804093567251462,
1289
+ "5": 0.6705882352941176,
1290
+ "6": 0.6466876971608833,
1291
+ "7": 0.5632911392405063,
1292
+ "8": 0.5525525525525525,
1293
+ "9": 0.5659163987138264,
1294
+ "10": 0.49850746268656715
1295
+ },
1296
+ "pass@128": {
1297
+ "2": 0.9857142857142858,
1298
+ "3": 0.9650145772594753,
1299
+ "4": 0.8128654970760234,
1300
+ "5": 0.6882352941176471,
1301
+ "6": 0.6561514195583596,
1302
+ "7": 0.5791139240506329,
1303
+ "8": 0.5615615615615616,
1304
+ "9": 0.5755627009646302,
1305
+ "10": 0.5104477611940299
1306
+ }
1307
+ },
1308
+ "total_pass_at_k": {
1309
+ "pass@1": 0.5754938064948109,
1310
+ "pass@2": 0.6072982926012722,
1311
+ "pass@4": 0.6340810177435554,
1312
+ "pass@8": 0.6551724137931034,
1313
+ "pass@16": 0.6702376966856378,
1314
+ "pass@32": 0.6856377636424507,
1315
+ "pass@64": 0.697689989956478,
1316
+ "pass@128": 0.7084030800133914
1317
+ }
1318
+ },
1319
+ "teachers_in_school": {
1320
+ "per_op_pass_at_k": {
1321
+ "2": {
1322
+ "pass@1": 0.9270516717325228,
1323
+ "pass@2": 0.9483282674772037,
1324
+ "pass@4": 0.9574468085106383,
1325
+ "pass@8": 0.9665653495440729,
1326
+ "pass@16": 0.9726443768996961,
1327
+ "pass@32": 0.9756838905775076,
1328
+ "pass@64": 0.9848024316109423,
1329
+ "pass@128": 0.9848024316109423
1330
+ },
1331
+ "3": {
1332
+ "pass@1": 0.8818181818181818,
1333
+ "pass@2": 0.9,
1334
+ "pass@4": 0.9151515151515152,
1335
+ "pass@8": 0.9363636363636364,
1336
+ "pass@16": 0.9363636363636364,
1337
+ "pass@32": 0.9454545454545454,
1338
+ "pass@64": 0.9575757575757575,
1339
+ "pass@128": 0.9666666666666667
1340
+ },
1341
+ "4": {
1342
+ "pass@1": 0.6568047337278107,
1343
+ "pass@2": 0.6775147928994083,
1344
+ "pass@4": 0.7071005917159763,
1345
+ "pass@8": 0.7307692307692307,
1346
+ "pass@16": 0.7514792899408284,
1347
+ "pass@32": 0.7603550295857988,
1348
+ "pass@64": 0.772189349112426,
1349
+ "pass@128": 0.7928994082840237
1350
+ },
1351
+ "5": {
1352
+ "pass@1": 0.5710144927536231,
1353
+ "pass@2": 0.6057971014492753,
1354
+ "pass@4": 0.6260869565217392,
1355
+ "pass@8": 0.6579710144927536,
1356
+ "pass@16": 0.6753623188405797,
1357
+ "pass@32": 0.7014492753623188,
1358
+ "pass@64": 0.7188405797101449,
1359
+ "pass@128": 0.7304347826086957
1360
+ },
1361
+ "6": {
1362
+ "pass@1": 0.4785100286532951,
1363
+ "pass@2": 0.49570200573065903,
1364
+ "pass@4": 0.5186246418338109,
1365
+ "pass@8": 0.5558739255014327,
1366
+ "pass@16": 0.5702005730659025,
1367
+ "pass@32": 0.5988538681948424,
1368
+ "pass@64": 0.6217765042979942,
1369
+ "pass@128": 0.6446991404011462
1370
+ },
1371
+ "7": {
1372
+ "pass@1": 0.47109826589595377,
1373
+ "pass@2": 0.4913294797687861,
1374
+ "pass@4": 0.5028901734104047,
1375
+ "pass@8": 0.5202312138728323,
1376
+ "pass@16": 0.5375722543352601,
1377
+ "pass@32": 0.5433526011560693,
1378
+ "pass@64": 0.5722543352601156,
1379
+ "pass@128": 0.5895953757225434
1380
+ },
1381
+ "8": {
1382
+ "pass@1": 0.43661971830985913,
1383
+ "pass@2": 0.4507042253521127,
1384
+ "pass@4": 0.48450704225352115,
1385
+ "pass@8": 0.504225352112676,
1386
+ "pass@16": 0.5098591549295775,
1387
+ "pass@32": 0.5211267605633803,
1388
+ "pass@64": 0.532394366197183,
1389
+ "pass@128": 0.5549295774647888
1390
+ },
1391
+ "9": {
1392
+ "pass@1": 0.4101796407185629,
1393
+ "pass@2": 0.4281437125748503,
1394
+ "pass@4": 0.45209580838323354,
1395
+ "pass@8": 0.47005988023952094,
1396
+ "pass@16": 0.47904191616766467,
1397
+ "pass@32": 0.48502994011976047,
1398
+ "pass@64": 0.5,
1399
+ "pass@128": 0.5269461077844312
1400
+ },
1401
+ "10": {
1402
+ "pass@1": 0.4785276073619632,
1403
+ "pass@2": 0.5,
1404
+ "pass@4": 0.50920245398773,
1405
+ "pass@8": 0.5122699386503068,
1406
+ "pass@16": 0.5184049079754601,
1407
+ "pass@32": 0.5368098159509203,
1408
+ "pass@64": 0.5398773006134969,
1409
+ "pass@128": 0.5460122699386503
1410
+ }
1411
+ },
1412
+ "per_op_pass_at_columns": {
1413
+ "pass@1": {
1414
+ "2": 0.9270516717325228,
1415
+ "3": 0.8818181818181818,
1416
+ "4": 0.6568047337278107,
1417
+ "5": 0.5710144927536231,
1418
+ "6": 0.4785100286532951,
1419
+ "7": 0.47109826589595377,
1420
+ "8": 0.43661971830985913,
1421
+ "9": 0.4101796407185629,
1422
+ "10": 0.4785276073619632
1423
+ },
1424
+ "pass@2": {
1425
+ "2": 0.9483282674772037,
1426
+ "3": 0.9,
1427
+ "4": 0.6775147928994083,
1428
+ "5": 0.6057971014492753,
1429
+ "6": 0.49570200573065903,
1430
+ "7": 0.4913294797687861,
1431
+ "8": 0.4507042253521127,
1432
+ "9": 0.4281437125748503,
1433
+ "10": 0.5
1434
+ },
1435
+ "pass@4": {
1436
+ "2": 0.9574468085106383,
1437
+ "3": 0.9151515151515152,
1438
+ "4": 0.7071005917159763,
1439
+ "5": 0.6260869565217392,
1440
+ "6": 0.5186246418338109,
1441
+ "7": 0.5028901734104047,
1442
+ "8": 0.48450704225352115,
1443
+ "9": 0.45209580838323354,
1444
+ "10": 0.50920245398773
1445
+ },
1446
+ "pass@8": {
1447
+ "2": 0.9665653495440729,
1448
+ "3": 0.9363636363636364,
1449
+ "4": 0.7307692307692307,
1450
+ "5": 0.6579710144927536,
1451
+ "6": 0.5558739255014327,
1452
+ "7": 0.5202312138728323,
1453
+ "8": 0.504225352112676,
1454
+ "9": 0.47005988023952094,
1455
+ "10": 0.5122699386503068
1456
+ },
1457
+ "pass@16": {
1458
+ "2": 0.9726443768996961,
1459
+ "3": 0.9363636363636364,
1460
+ "4": 0.7514792899408284,
1461
+ "5": 0.6753623188405797,
1462
+ "6": 0.5702005730659025,
1463
+ "7": 0.5375722543352601,
1464
+ "8": 0.5098591549295775,
1465
+ "9": 0.47904191616766467,
1466
+ "10": 0.5184049079754601
1467
+ },
1468
+ "pass@32": {
1469
+ "2": 0.9756838905775076,
1470
+ "3": 0.9454545454545454,
1471
+ "4": 0.7603550295857988,
1472
+ "5": 0.7014492753623188,
1473
+ "6": 0.5988538681948424,
1474
+ "7": 0.5433526011560693,
1475
+ "8": 0.5211267605633803,
1476
+ "9": 0.48502994011976047,
1477
+ "10": 0.5368098159509203
1478
+ },
1479
+ "pass@64": {
1480
+ "2": 0.9848024316109423,
1481
+ "3": 0.9575757575757575,
1482
+ "4": 0.772189349112426,
1483
+ "5": 0.7188405797101449,
1484
+ "6": 0.6217765042979942,
1485
+ "7": 0.5722543352601156,
1486
+ "8": 0.532394366197183,
1487
+ "9": 0.5,
1488
+ "10": 0.5398773006134969
1489
+ },
1490
+ "pass@128": {
1491
+ "2": 0.9848024316109423,
1492
+ "3": 0.9666666666666667,
1493
+ "4": 0.7928994082840237,
1494
+ "5": 0.7304347826086957,
1495
+ "6": 0.6446991404011462,
1496
+ "7": 0.5895953757225434,
1497
+ "8": 0.5549295774647888,
1498
+ "9": 0.5269461077844312,
1499
+ "10": 0.5460122699386503
1500
+ }
1501
+ },
1502
+ "total_pass_at_k": {
1503
+ "pass@1": 0.5874836173001311,
1504
+ "pass@2": 0.6081258191349934,
1505
+ "pass@4": 0.6277850589777195,
1506
+ "pass@8": 0.6480996068152032,
1507
+ "pass@16": 0.6589121887287025,
1508
+ "pass@32": 0.6720183486238532,
1509
+ "pass@64": 0.6867627785058977,
1510
+ "pass@128": 0.7021625163826999
1511
+ }
1512
+ }
1513
+ },
1514
+ "total": {
1515
+ "per_op_pass_at_k": {
1516
+ "2": {
1517
+ "pass@1": 0.915,
1518
+ "pass@2": 0.936,
1519
+ "pass@4": 0.954,
1520
+ "pass@8": 0.962,
1521
+ "pass@16": 0.974,
1522
+ "pass@32": 0.978,
1523
+ "pass@64": 0.983,
1524
+ "pass@128": 0.986
1525
+ },
1526
+ "3": {
1527
+ "pass@1": 0.871,
1528
+ "pass@2": 0.894,
1529
+ "pass@4": 0.912,
1530
+ "pass@8": 0.933,
1531
+ "pass@16": 0.937,
1532
+ "pass@32": 0.947,
1533
+ "pass@64": 0.959,
1534
+ "pass@128": 0.968
1535
+ },
1536
+ "4": {
1537
+ "pass@1": 0.667,
1538
+ "pass@2": 0.694,
1539
+ "pass@4": 0.718,
1540
+ "pass@8": 0.737,
1541
+ "pass@16": 0.754,
1542
+ "pass@32": 0.77,
1543
+ "pass@64": 0.783,
1544
+ "pass@128": 0.795
1545
+ },
1546
+ "5": {
1547
+ "pass@1": 0.545,
1548
+ "pass@2": 0.578,
1549
+ "pass@4": 0.602,
1550
+ "pass@8": 0.626,
1551
+ "pass@16": 0.651,
1552
+ "pass@32": 0.675,
1553
+ "pass@64": 0.694,
1554
+ "pass@128": 0.708
1555
+ },
1556
+ "6": {
1557
+ "pass@1": 0.483,
1558
+ "pass@2": 0.513,
1559
+ "pass@4": 0.545,
1560
+ "pass@8": 0.576,
1561
+ "pass@16": 0.598,
1562
+ "pass@32": 0.615,
1563
+ "pass@64": 0.634,
1564
+ "pass@128": 0.656
1565
+ },
1566
+ "7": {
1567
+ "pass@1": 0.462,
1568
+ "pass@2": 0.49,
1569
+ "pass@4": 0.509,
1570
+ "pass@8": 0.536,
1571
+ "pass@16": 0.558,
1572
+ "pass@32": 0.57,
1573
+ "pass@64": 0.587,
1574
+ "pass@128": 0.601
1575
+ },
1576
+ "8": {
1577
+ "pass@1": 0.433,
1578
+ "pass@2": 0.453,
1579
+ "pass@4": 0.479,
1580
+ "pass@8": 0.497,
1581
+ "pass@16": 0.506,
1582
+ "pass@32": 0.532,
1583
+ "pass@64": 0.545,
1584
+ "pass@128": 0.559
1585
+ },
1586
+ "9": {
1587
+ "pass@1": 0.427,
1588
+ "pass@2": 0.456,
1589
+ "pass@4": 0.49,
1590
+ "pass@8": 0.506,
1591
+ "pass@16": 0.521,
1592
+ "pass@32": 0.533,
1593
+ "pass@64": 0.548,
1594
+ "pass@128": 0.57
1595
+ },
1596
+ "10": {
1597
+ "pass@1": 0.455,
1598
+ "pass@2": 0.475,
1599
+ "pass@4": 0.49,
1600
+ "pass@8": 0.501,
1601
+ "pass@16": 0.507,
1602
+ "pass@32": 0.519,
1603
+ "pass@64": 0.524,
1604
+ "pass@128": 0.532
1605
+ }
1606
+ },
1607
+ "per_op_pass_at_columns": {
1608
+ "pass@1": {
1609
+ "2": 0.915,
1610
+ "3": 0.871,
1611
+ "4": 0.667,
1612
+ "5": 0.545,
1613
+ "6": 0.483,
1614
+ "7": 0.462,
1615
+ "8": 0.433,
1616
+ "9": 0.427,
1617
+ "10": 0.455
1618
+ },
1619
+ "pass@2": {
1620
+ "2": 0.936,
1621
+ "3": 0.894,
1622
+ "4": 0.694,
1623
+ "5": 0.578,
1624
+ "6": 0.513,
1625
+ "7": 0.49,
1626
+ "8": 0.453,
1627
+ "9": 0.456,
1628
+ "10": 0.475
1629
+ },
1630
+ "pass@4": {
1631
+ "2": 0.954,
1632
+ "3": 0.912,
1633
+ "4": 0.718,
1634
+ "5": 0.602,
1635
+ "6": 0.545,
1636
+ "7": 0.509,
1637
+ "8": 0.479,
1638
+ "9": 0.49,
1639
+ "10": 0.49
1640
+ },
1641
+ "pass@8": {
1642
+ "2": 0.962,
1643
+ "3": 0.933,
1644
+ "4": 0.737,
1645
+ "5": 0.626,
1646
+ "6": 0.576,
1647
+ "7": 0.536,
1648
+ "8": 0.497,
1649
+ "9": 0.506,
1650
+ "10": 0.501
1651
+ },
1652
+ "pass@16": {
1653
+ "2": 0.974,
1654
+ "3": 0.937,
1655
+ "4": 0.754,
1656
+ "5": 0.651,
1657
+ "6": 0.598,
1658
+ "7": 0.558,
1659
+ "8": 0.506,
1660
+ "9": 0.521,
1661
+ "10": 0.507
1662
+ },
1663
+ "pass@32": {
1664
+ "2": 0.978,
1665
+ "3": 0.947,
1666
+ "4": 0.77,
1667
+ "5": 0.675,
1668
+ "6": 0.615,
1669
+ "7": 0.57,
1670
+ "8": 0.532,
1671
+ "9": 0.533,
1672
+ "10": 0.519
1673
+ },
1674
+ "pass@64": {
1675
+ "2": 0.983,
1676
+ "3": 0.959,
1677
+ "4": 0.783,
1678
+ "5": 0.694,
1679
+ "6": 0.634,
1680
+ "7": 0.587,
1681
+ "8": 0.545,
1682
+ "9": 0.548,
1683
+ "10": 0.524
1684
+ },
1685
+ "pass@128": {
1686
+ "2": 0.986,
1687
+ "3": 0.968,
1688
+ "4": 0.795,
1689
+ "5": 0.708,
1690
+ "6": 0.656,
1691
+ "7": 0.601,
1692
+ "8": 0.559,
1693
+ "9": 0.57,
1694
+ "10": 0.532
1695
+ }
1696
+ },
1697
+ "total_pass_at_k": {
1698
+ "pass@1": 0.5842222222222222,
1699
+ "pass@2": 0.6098888888888889,
1700
+ "pass@4": 0.6332222222222222,
1701
+ "pass@8": 0.6526666666666666,
1702
+ "pass@16": 0.6673333333333333,
1703
+ "pass@32": 0.6821111111111111,
1704
+ "pass@64": 0.6952222222222222,
1705
+ "pass@128": 0.7083333333333334
1706
+ }
1707
+ },
1708
+ "ks": [
1709
+ 1,
1710
+ 2,
1711
+ 4,
1712
+ 8,
1713
+ 16,
1714
+ 32,
1715
+ 64,
1716
+ 128
1717
+ ],
1718
+ "prompt_counts": {
1719
+ "per_template": {
1720
+ "crazy_zootopia": {
1721
+ "per_op": {
1722
+ "2": 321,
1723
+ "3": 327,
1724
+ "4": 320,
1725
+ "5": 315,
1726
+ "6": 334,
1727
+ "7": 338,
1728
+ "8": 312,
1729
+ "9": 355,
1730
+ "10": 339
1731
+ },
1732
+ "total": 2961
1733
+ },
1734
+ "movie_festival_awards": {
1735
+ "per_op": {
1736
+ "2": 350,
1737
+ "3": 343,
1738
+ "4": 342,
1739
+ "5": 340,
1740
+ "6": 317,
1741
+ "7": 316,
1742
+ "8": 333,
1743
+ "9": 311,
1744
+ "10": 335
1745
+ },
1746
+ "total": 2987
1747
+ },
1748
+ "teachers_in_school": {
1749
+ "per_op": {
1750
+ "2": 329,
1751
+ "3": 330,
1752
+ "4": 338,
1753
+ "5": 345,
1754
+ "6": 349,
1755
+ "7": 346,
1756
+ "8": 355,
1757
+ "9": 334,
1758
+ "10": 326
1759
+ },
1760
+ "total": 3052
1761
+ }
1762
+ },
1763
+ "total": {
1764
+ "per_op": {
1765
+ "2": 1000,
1766
+ "3": 1000,
1767
+ "4": 1000,
1768
+ "5": 1000,
1769
+ "6": 1000,
1770
+ "7": 1000,
1771
+ "8": 1000,
1772
+ "9": 1000,
1773
+ "10": 1000
1774
+ },
1775
+ "total": 9000
1776
+ }
1777
+ }
1778
+ }
1779
+ }
1780
+ }