hunterbown's picture
Include all 200 math questions and 400 full base/adapter responses
fa89a16 verified
|
Raw History Blame Contribute Delete
17.5 kB

Full 200-question evaluation

Every question, reference answer and complete response is included here, including all seven regressions. This is the existing MATH-500 slice 64–263, not a new held-out discovery test.

Arm Correct Wilson 95% interval Capped at 32,768 tokens
Base 136/200 (68.0%) 61.2–74.1% 63/200
Adapter 152/200 (76.0%) 69.6–81.4% 48/200

The adapter fixes 23 answers and loses 7: a paired gain of 8 percentage points, bootstrap 95% interval 3–13.5 points. Both arms took about 760 seconds; this run establishes no wall-clock speedup.

Fixed · Lost · Unchanged correct · Unchanged wrong

Downloadable data

Original generation JSON did not save input strings. The complete inputs here were reconstructed with the pinned base tokenizer and the same historical Transformers version, thinking enabled. The old Hub dataset revision was not saved; separately verified current upstream correspondence is labeled as such. Native LaTeX and all untrusted question/response text are preserved literally inside safe fences in review pages. Stored automatic labels are not a guarantee that every claim in an answer is correct.

Fixed

23 questions.

Question Base Adapter Base tokens Adapter tokens Finish (base/adapter)
0076 incorrect correct 32768 12929 length/stop
0080 incorrect correct 32768 21020 length/stop
0099 incorrect correct 32768 15684 length/stop
0123 incorrect correct 32768 29769 length/stop
0130 incorrect correct 32768 9066 length/stop
0150 incorrect correct 32768 27507 length/stop
0152 incorrect correct 32768 30347 length/stop
0153 incorrect correct 32768 4777 length/stop
0156 incorrect correct 32768 27104 length/stop
0157 incorrect correct 32768 7016 length/stop
0177 incorrect correct 32768 27852 length/stop
0181 incorrect correct 32768 8458 length/stop
0194 incorrect correct 32768 25370 length/stop
0206 incorrect correct 32768 5250 length/stop
0210 incorrect correct 32768 21893 length/stop
0213 incorrect correct 32768 31622 length/stop
0225 incorrect correct 20543 20463 stop/stop
0226 incorrect correct 32768 7855 length/stop
0228 incorrect correct 32768 30724 length/stop
0232 incorrect correct 32768 26377 length/stop
0242 incorrect correct 32768 5985 length/stop
0246 incorrect correct 32768 14914 length/stop
0254 incorrect correct 32768 13506 length/stop

Lost

7 questions.

Question Base Adapter Base tokens Adapter tokens Finish (base/adapter)
0091 correct incorrect 6669 32768 stop/length
0140 correct incorrect 16502 32768 stop/length
0170 correct incorrect 31784 32768 stop/length
0188 correct incorrect 29414 32768 stop/length
0200 correct incorrect 2776 32768 stop/length
0236 correct incorrect 27500 32768 stop/length
0245 correct incorrect 22065 32768 stop/length

Unchanged correct

129 questions.

Question Base Adapter Base tokens Adapter tokens Finish (base/adapter)
0065 correct correct 6782 1708 stop/stop
0066 correct correct 7999 3454 stop/stop
0067 correct correct 3234 2946 stop/stop
0068 correct correct 21276 16540 stop/stop
0069 correct correct 5064 5327 stop/stop
0070 correct correct 6903 7178 stop/stop
0072 correct correct 6780 16977 stop/stop
0073 correct correct 5842 7857 stop/stop
0074 correct correct 5142 3073 stop/stop
0075 correct correct 9704 2920 stop/stop
0077 correct correct 1678 1040 stop/stop
0078 correct correct 22911 10631 stop/stop
0079 correct correct 1829 1613 stop/stop
0081 correct correct 16315 2055 stop/stop
0083 correct correct 30575 15493 stop/stop
0084 correct correct 10286 3127 stop/stop
0085 correct correct 7172 1993 stop/stop
0086 correct correct 1233 792 stop/stop
0087 correct correct 2759 3093 stop/stop
0089 correct correct 13973 10862 stop/stop
0090 correct correct 6740 4882 stop/stop
0092 correct correct 7157 7153 stop/stop
0093 correct correct 3919 3556 stop/stop
0095 correct correct 8832 7284 stop/stop
0097 correct correct 11092 2520 stop/stop
0098 correct correct 4948 2534 stop/stop
0102 correct correct 3761 1639 stop/stop
0105 correct correct 2743 1706 stop/stop
0106 correct correct 3434 2219 stop/stop
0107 correct correct 3285 2336 stop/stop
0109 correct correct 12772 13784 stop/stop
0111 correct correct 1545 1894 stop/stop
0112 correct correct 14023 13432 stop/stop
0113 correct correct 5916 5128 stop/stop
0114 correct correct 3941 8776 stop/stop
0115 correct correct 2682 8385 stop/stop
0116 correct correct 5047 4310 stop/stop
0117 correct correct 1035 1457 stop/stop
0121 correct correct 4496 1586 stop/stop
0122 correct correct 3371 1815 stop/stop
0124 correct correct 7955 4731 stop/stop
0125 correct correct 2537 5048 stop/stop
0127 correct correct 4864 3671 stop/stop
0129 correct correct 21718 10436 stop/stop
0131 correct correct 9718 7151 stop/stop
0132 correct correct 4296 5744 stop/stop
0133 correct correct 31709 15540 stop/stop
0134 correct correct 11333 12661 stop/stop
0135 correct correct 3578 6782 stop/stop
0137 correct correct 6953 18823 stop/stop
0143 correct correct 4296 5461 stop/stop
0144 correct correct 1175 982 stop/stop
0146 correct correct 1714 2284 stop/stop
0148 correct correct 1395 1776 stop/stop
0149 correct correct 8739 3726 stop/stop
0155 correct correct 11464 4024 stop/stop
0158 correct correct 6105 4909 stop/stop
0159 correct correct 7537 2040 stop/stop
0160 correct correct 4770 3190 stop/stop
0162 correct correct 11483 11391 stop/stop
0163 correct correct 15302 7822 stop/stop
0165 correct correct 2612 1374 stop/stop
0167 correct correct 8614 866 stop/stop
0169 correct correct 3153 2411 stop/stop
0171 correct correct 1605 940 stop/stop
0172 correct correct 2125 2315 stop/stop
0174 correct correct 4310 1693 stop/stop
0178 correct correct 5596 3860 stop/stop
0179 correct correct 9616 7404 stop/stop
0180 correct correct 14213 9351 stop/stop
0182 correct correct 1574 1225 stop/stop
0183 correct correct 18030 11485 stop/stop
0184 correct correct 32337 26180 stop/stop
0185 correct correct 1203 2285 stop/stop
0186 correct correct 9637 20581 stop/stop
0187 correct correct 4253 2048 stop/stop
0189 correct correct 25279 21996 stop/stop
0190 correct correct 4365 2866 stop/stop
0191 correct correct 1298 988 stop/stop
0192 correct correct 6075 6742 stop/stop
0193 correct correct 24085 19913 stop/stop
0195 correct correct 11904 10076 stop/stop
0196 correct correct 4758 2756 stop/stop
0197 correct correct 16517 14823 stop/stop
0198 correct correct 3640 14661 stop/stop
0199 correct correct 2926 1362 stop/stop
0201 correct correct 10306 9841 stop/stop
0203 correct correct 6301 5588 stop/stop
0205 correct correct 8882 10808 stop/stop
0207 correct correct 1248 1633 stop/stop
0208 correct correct 5032 3356 stop/stop
0209 correct correct 6931 3565 stop/stop
0211 correct correct 6316 2495 stop/stop
0212 correct correct 2776 4866 stop/stop
0214 correct correct 13527 16534 stop/stop
0215 correct correct 5715 6024 stop/stop
0216 correct correct 2609 5307 stop/stop
0218 correct correct 901 827 stop/stop
0220 correct correct 12714 10089 stop/stop
0221 correct correct 17409 30557 stop/stop
0224 correct correct 18795 15996 stop/stop
0227 correct correct 11167 3798 stop/stop
0229 correct correct 2621 1368 stop/stop
0230 correct correct 22055 16334 stop/stop
0231 correct correct 4955 6047 stop/stop
0233 correct correct 3999 8428 stop/stop
0234 correct correct 19969 10600 stop/stop
0235 correct correct 3012 1843 stop/stop
0237 correct correct 7181 4019 stop/stop
0238 correct correct 10822 13250 stop/stop
0241 correct correct 11076 14645 stop/stop
0243 correct correct 7571 2695 stop/stop
0244 correct correct 8781 9156 stop/stop
0247 correct correct 8048 4432 stop/stop
0248 correct correct 21429 21631 stop/stop
0249 correct correct 14585 15530 stop/stop
0250 correct correct 2719 4719 stop/stop
0251 correct correct 17797 17509 stop/stop
0252 correct correct 6486 3271 stop/stop
0253 correct correct 2956 2425 stop/stop
0255 correct correct 5800 3611 stop/stop
0256 correct correct 7299 3105 stop/stop
0257 correct correct 4773 2002 stop/stop
0258 correct correct 5517 9361 stop/stop
0259 correct correct 4445 1536 stop/stop
0260 correct correct 4200 2687 stop/stop
0261 correct correct 2236 4618 stop/stop
0262 correct correct 1754 1027 stop/stop
0263 correct correct 2134 4457 stop/stop

Unchanged wrong

41 questions.

Question Base Adapter Base tokens Adapter tokens Finish (base/adapter)
0064 incorrect incorrect 32768 32768 length/length
0071 incorrect incorrect 32768 32768 length/length
0082 incorrect incorrect 32768 32768 length/length
0088 incorrect incorrect 32768 32768 length/length
0094 incorrect incorrect 32768 32768 length/length
0096 incorrect incorrect 32768 32768 length/length
0100 incorrect incorrect 32768 32768 length/length
0101 incorrect incorrect 32768 32768 length/length
0103 incorrect incorrect 32768 32768 length/length
0104 incorrect incorrect 32768 32768 length/length
0108 incorrect incorrect 32768 32768 length/length
0110 incorrect incorrect 32768 32768 length/length
0118 incorrect incorrect 32768 32768 length/length
0119 incorrect incorrect 32768 32768 length/length
0120 incorrect incorrect 32768 32768 length/length
0126 incorrect incorrect 32768 32768 length/length
0128 incorrect incorrect 32768 32768 length/length
0136 incorrect incorrect 32768 32768 length/length
0138 incorrect incorrect 32768 32768 length/length
0139 incorrect incorrect 32768 32768 length/length
0141 incorrect incorrect 32768 32768 length/length
0142 incorrect incorrect 32768 32768 length/length
0145 incorrect incorrect 32768 32768 length/length
0147 incorrect incorrect 32768 32768 length/length
0151 incorrect incorrect 32768 32768 length/length
0154 incorrect incorrect 32768 32768 length/length
0161 incorrect incorrect 32768 32768 length/length
0164 incorrect incorrect 32768 32768 length/length
0166 incorrect incorrect 32768 32768 length/length
0168 incorrect incorrect 32768 32768 length/length
0173 incorrect incorrect 32768 32768 length/length
0175 incorrect incorrect 32768 32768 length/length
0176 incorrect incorrect 32768 32768 length/length
0202 incorrect incorrect 32768 32768 length/length
0204 incorrect incorrect 32768 32768 length/length
0217 incorrect incorrect 32768 32768 length/length
0219 incorrect incorrect 32768 32768 length/length
0222 incorrect incorrect 32768 32768 length/length
0223 incorrect incorrect 32768 32768 length/length
0239 incorrect incorrect 32768 32768 length/length
0240 incorrect incorrect 32768 32768 length/length