# Full 200-question evaluation Every question, reference answer and complete response is included here, including all seven regressions. This is the existing MATH-500 slice 64–263, not a new held-out discovery test. | Arm | Correct | Wilson 95% interval | Capped at 32,768 tokens | |---|---:|---:|---:| | Base | 136/200 (68.0%) | 61.2–74.1% | 63/200 | | Adapter | 152/200 (76.0%) | 69.6–81.4% | 48/200 | The adapter fixes 23 answers and loses 7: a paired gain of 8 percentage points, bootstrap 95% interval 3–13.5 points. Both arms took about 760 seconds; this run establishes no wall-clock speedup. [Fixed](#fixed) · [Lost](#lost) · [Unchanged correct](#unchanged-correct) · [Unchanged wrong](#unchanged-wrong) ## Downloadable data - [questions.jsonl](questions.jsonl): all 200 questions, reference answers, IDs/order and full reconstructed input strings. - [responses.jsonl.gz](responses.jsonl.gz): both full outputs and original correct/finish/token flags, with no truncation. - [summary.csv](summary.csv): spreadsheet-safe overview; potentially executable text cells are prefixed with an apostrophe. JSONL retains exact original strings. - [protocol.json](protocol.json): full settings, runtime, model/data/source hashes and scope. - [Attribution](ATTRIBUTION.md) and [dataset license](DATASET_LICENSE). - [File manifest](manifest.json). Original generation JSON did not save input strings. The complete inputs here were reconstructed with the pinned base tokenizer and the same historical Transformers version, thinking enabled. The old Hub dataset revision was not saved; separately verified current upstream correspondence is labeled as such. Native LaTeX and all untrusted question/response text are preserved literally inside safe fences in review pages. Stored automatic labels are not a guarantee that every claim in an answer is correct. ## Fixed 23 questions. | Question | Base | Adapter | Base tokens | Adapter tokens | Finish (base/adapter) | |---|---|---|---:|---:|---| | [0076](items/0076.md) | incorrect | correct | 32768 | 12929 | length/stop | | [0080](items/0080.md) | incorrect | correct | 32768 | 21020 | length/stop | | [0099](items/0099.md) | incorrect | correct | 32768 | 15684 | length/stop | | [0123](items/0123.md) | incorrect | correct | 32768 | 29769 | length/stop | | [0130](items/0130.md) | incorrect | correct | 32768 | 9066 | length/stop | | [0150](items/0150.md) | incorrect | correct | 32768 | 27507 | length/stop | | [0152](items/0152.md) | incorrect | correct | 32768 | 30347 | length/stop | | [0153](items/0153.md) | incorrect | correct | 32768 | 4777 | length/stop | | [0156](items/0156.md) | incorrect | correct | 32768 | 27104 | length/stop | | [0157](items/0157.md) | incorrect | correct | 32768 | 7016 | length/stop | | [0177](items/0177.md) | incorrect | correct | 32768 | 27852 | length/stop | | [0181](items/0181.md) | incorrect | correct | 32768 | 8458 | length/stop | | [0194](items/0194.md) | incorrect | correct | 32768 | 25370 | length/stop | | [0206](items/0206.md) | incorrect | correct | 32768 | 5250 | length/stop | | [0210](items/0210.md) | incorrect | correct | 32768 | 21893 | length/stop | | [0213](items/0213.md) | incorrect | correct | 32768 | 31622 | length/stop | | [0225](items/0225.md) | incorrect | correct | 20543 | 20463 | stop/stop | | [0226](items/0226.md) | incorrect | correct | 32768 | 7855 | length/stop | | [0228](items/0228.md) | incorrect | correct | 32768 | 30724 | length/stop | | [0232](items/0232.md) | incorrect | correct | 32768 | 26377 | length/stop | | [0242](items/0242.md) | incorrect | correct | 32768 | 5985 | length/stop | | [0246](items/0246.md) | incorrect | correct | 32768 | 14914 | length/stop | | [0254](items/0254.md) | incorrect | correct | 32768 | 13506 | length/stop | ## Lost 7 questions. | Question | Base | Adapter | Base tokens | Adapter tokens | Finish (base/adapter) | |---|---|---|---:|---:|---| | [0091](items/0091.md) | correct | incorrect | 6669 | 32768 | stop/length | | [0140](items/0140.md) | correct | incorrect | 16502 | 32768 | stop/length | | [0170](items/0170.md) | correct | incorrect | 31784 | 32768 | stop/length | | [0188](items/0188.md) | correct | incorrect | 29414 | 32768 | stop/length | | [0200](items/0200.md) | correct | incorrect | 2776 | 32768 | stop/length | | [0236](items/0236.md) | correct | incorrect | 27500 | 32768 | stop/length | | [0245](items/0245.md) | correct | incorrect | 22065 | 32768 | stop/length | ## Unchanged correct 129 questions. | Question | Base | Adapter | Base tokens | Adapter tokens | Finish (base/adapter) | |---|---|---|---:|---:|---| | [0065](items/0065.md) | correct | correct | 6782 | 1708 | stop/stop | | [0066](items/0066.md) | correct | correct | 7999 | 3454 | stop/stop | | [0067](items/0067.md) | correct | correct | 3234 | 2946 | stop/stop | | [0068](items/0068.md) | correct | correct | 21276 | 16540 | stop/stop | | [0069](items/0069.md) | correct | correct | 5064 | 5327 | stop/stop | | [0070](items/0070.md) | correct | correct | 6903 | 7178 | stop/stop | | [0072](items/0072.md) | correct | correct | 6780 | 16977 | stop/stop | | [0073](items/0073.md) | correct | correct | 5842 | 7857 | stop/stop | | [0074](items/0074.md) | correct | correct | 5142 | 3073 | stop/stop | | [0075](items/0075.md) | correct | correct | 9704 | 2920 | stop/stop | | [0077](items/0077.md) | correct | correct | 1678 | 1040 | stop/stop | | [0078](items/0078.md) | correct | correct | 22911 | 10631 | stop/stop | | [0079](items/0079.md) | correct | correct | 1829 | 1613 | stop/stop | | [0081](items/0081.md) | correct | correct | 16315 | 2055 | stop/stop | | [0083](items/0083.md) | correct | correct | 30575 | 15493 | stop/stop | | [0084](items/0084.md) | correct | correct | 10286 | 3127 | stop/stop | | [0085](items/0085.md) | correct | correct | 7172 | 1993 | stop/stop | | [0086](items/0086.md) | correct | correct | 1233 | 792 | stop/stop | | [0087](items/0087.md) | correct | correct | 2759 | 3093 | stop/stop | | [0089](items/0089.md) | correct | correct | 13973 | 10862 | stop/stop | | [0090](items/0090.md) | correct | correct | 6740 | 4882 | stop/stop | | [0092](items/0092.md) | correct | correct | 7157 | 7153 | stop/stop | | [0093](items/0093.md) | correct | correct | 3919 | 3556 | stop/stop | | [0095](items/0095.md) | correct | correct | 8832 | 7284 | stop/stop | | [0097](items/0097.md) | correct | correct | 11092 | 2520 | stop/stop | | [0098](items/0098.md) | correct | correct | 4948 | 2534 | stop/stop | | [0102](items/0102.md) | correct | correct | 3761 | 1639 | stop/stop | | [0105](items/0105.md) | correct | correct | 2743 | 1706 | stop/stop | | [0106](items/0106.md) | correct | correct | 3434 | 2219 | stop/stop | | [0107](items/0107.md) | correct | correct | 3285 | 2336 | stop/stop | | [0109](items/0109.md) | correct | correct | 12772 | 13784 | stop/stop | | [0111](items/0111.md) | correct | correct | 1545 | 1894 | stop/stop | | [0112](items/0112.md) | correct | correct | 14023 | 13432 | stop/stop | | [0113](items/0113.md) | correct | correct | 5916 | 5128 | stop/stop | | [0114](items/0114.md) | correct | correct | 3941 | 8776 | stop/stop | | [0115](items/0115.md) | correct | correct | 2682 | 8385 | stop/stop | | [0116](items/0116.md) | correct | correct | 5047 | 4310 | stop/stop | | [0117](items/0117.md) | correct | correct | 1035 | 1457 | stop/stop | | [0121](items/0121.md) | correct | correct | 4496 | 1586 | stop/stop | | [0122](items/0122.md) | correct | correct | 3371 | 1815 | stop/stop | | [0124](items/0124.md) | correct | correct | 7955 | 4731 | stop/stop | | [0125](items/0125.md) | correct | correct | 2537 | 5048 | stop/stop | | [0127](items/0127.md) | correct | correct | 4864 | 3671 | stop/stop | | [0129](items/0129.md) | correct | correct | 21718 | 10436 | stop/stop | | [0131](items/0131.md) | correct | correct | 9718 | 7151 | stop/stop | | [0132](items/0132.md) | correct | correct | 4296 | 5744 | stop/stop | | [0133](items/0133.md) | correct | correct | 31709 | 15540 | stop/stop | | [0134](items/0134.md) | correct | correct | 11333 | 12661 | stop/stop | | [0135](items/0135.md) | correct | correct | 3578 | 6782 | stop/stop | | [0137](items/0137.md) | correct | correct | 6953 | 18823 | stop/stop | | [0143](items/0143.md) | correct | correct | 4296 | 5461 | stop/stop | | [0144](items/0144.md) | correct | correct | 1175 | 982 | stop/stop | | [0146](items/0146.md) | correct | correct | 1714 | 2284 | stop/stop | | [0148](items/0148.md) | correct | correct | 1395 | 1776 | stop/stop | | [0149](items/0149.md) | correct | correct | 8739 | 3726 | stop/stop | | [0155](items/0155.md) | correct | correct | 11464 | 4024 | stop/stop | | [0158](items/0158.md) | correct | correct | 6105 | 4909 | stop/stop | | [0159](items/0159.md) | correct | correct | 7537 | 2040 | stop/stop | | [0160](items/0160.md) | correct | correct | 4770 | 3190 | stop/stop | | [0162](items/0162.md) | correct | correct | 11483 | 11391 | stop/stop | | [0163](items/0163.md) | correct | correct | 15302 | 7822 | stop/stop | | [0165](items/0165.md) | correct | correct | 2612 | 1374 | stop/stop | | [0167](items/0167.md) | correct | correct | 8614 | 866 | stop/stop | | [0169](items/0169.md) | correct | correct | 3153 | 2411 | stop/stop | | [0171](items/0171.md) | correct | correct | 1605 | 940 | stop/stop | | [0172](items/0172.md) | correct | correct | 2125 | 2315 | stop/stop | | [0174](items/0174.md) | correct | correct | 4310 | 1693 | stop/stop | | [0178](items/0178.md) | correct | correct | 5596 | 3860 | stop/stop | | [0179](items/0179.md) | correct | correct | 9616 | 7404 | stop/stop | | [0180](items/0180.md) | correct | correct | 14213 | 9351 | stop/stop | | [0182](items/0182.md) | correct | correct | 1574 | 1225 | stop/stop | | [0183](items/0183.md) | correct | correct | 18030 | 11485 | stop/stop | | [0184](items/0184.md) | correct | correct | 32337 | 26180 | stop/stop | | [0185](items/0185.md) | correct | correct | 1203 | 2285 | stop/stop | | [0186](items/0186.md) | correct | correct | 9637 | 20581 | stop/stop | | [0187](items/0187.md) | correct | correct | 4253 | 2048 | stop/stop | | [0189](items/0189.md) | correct | correct | 25279 | 21996 | stop/stop | | [0190](items/0190.md) | correct | correct | 4365 | 2866 | stop/stop | | [0191](items/0191.md) | correct | correct | 1298 | 988 | stop/stop | | [0192](items/0192.md) | correct | correct | 6075 | 6742 | stop/stop | | [0193](items/0193.md) | correct | correct | 24085 | 19913 | stop/stop | | [0195](items/0195.md) | correct | correct | 11904 | 10076 | stop/stop | | [0196](items/0196.md) | correct | correct | 4758 | 2756 | stop/stop | | [0197](items/0197.md) | correct | correct | 16517 | 14823 | stop/stop | | [0198](items/0198.md) | correct | correct | 3640 | 14661 | stop/stop | | [0199](items/0199.md) | correct | correct | 2926 | 1362 | stop/stop | | [0201](items/0201.md) | correct | correct | 10306 | 9841 | stop/stop | | [0203](items/0203.md) | correct | correct | 6301 | 5588 | stop/stop | | [0205](items/0205.md) | correct | correct | 8882 | 10808 | stop/stop | | [0207](items/0207.md) | correct | correct | 1248 | 1633 | stop/stop | | [0208](items/0208.md) | correct | correct | 5032 | 3356 | stop/stop | | [0209](items/0209.md) | correct | correct | 6931 | 3565 | stop/stop | | [0211](items/0211.md) | correct | correct | 6316 | 2495 | stop/stop | | [0212](items/0212.md) | correct | correct | 2776 | 4866 | stop/stop | | [0214](items/0214.md) | correct | correct | 13527 | 16534 | stop/stop | | [0215](items/0215.md) | correct | correct | 5715 | 6024 | stop/stop | | [0216](items/0216.md) | correct | correct | 2609 | 5307 | stop/stop | | [0218](items/0218.md) | correct | correct | 901 | 827 | stop/stop | | [0220](items/0220.md) | correct | correct | 12714 | 10089 | stop/stop | | [0221](items/0221.md) | correct | correct | 17409 | 30557 | stop/stop | | [0224](items/0224.md) | correct | correct | 18795 | 15996 | stop/stop | | [0227](items/0227.md) | correct | correct | 11167 | 3798 | stop/stop | | [0229](items/0229.md) | correct | correct | 2621 | 1368 | stop/stop | | [0230](items/0230.md) | correct | correct | 22055 | 16334 | stop/stop | | [0231](items/0231.md) | correct | correct | 4955 | 6047 | stop/stop | | [0233](items/0233.md) | correct | correct | 3999 | 8428 | stop/stop | | [0234](items/0234.md) | correct | correct | 19969 | 10600 | stop/stop | | [0235](items/0235.md) | correct | correct | 3012 | 1843 | stop/stop | | [0237](items/0237.md) | correct | correct | 7181 | 4019 | stop/stop | | [0238](items/0238.md) | correct | correct | 10822 | 13250 | stop/stop | | [0241](items/0241.md) | correct | correct | 11076 | 14645 | stop/stop | | [0243](items/0243.md) | correct | correct | 7571 | 2695 | stop/stop | | [0244](items/0244.md) | correct | correct | 8781 | 9156 | stop/stop | | [0247](items/0247.md) | correct | correct | 8048 | 4432 | stop/stop | | [0248](items/0248.md) | correct | correct | 21429 | 21631 | stop/stop | | [0249](items/0249.md) | correct | correct | 14585 | 15530 | stop/stop | | [0250](items/0250.md) | correct | correct | 2719 | 4719 | stop/stop | | [0251](items/0251.md) | correct | correct | 17797 | 17509 | stop/stop | | [0252](items/0252.md) | correct | correct | 6486 | 3271 | stop/stop | | [0253](items/0253.md) | correct | correct | 2956 | 2425 | stop/stop | | [0255](items/0255.md) | correct | correct | 5800 | 3611 | stop/stop | | [0256](items/0256.md) | correct | correct | 7299 | 3105 | stop/stop | | [0257](items/0257.md) | correct | correct | 4773 | 2002 | stop/stop | | [0258](items/0258.md) | correct | correct | 5517 | 9361 | stop/stop | | [0259](items/0259.md) | correct | correct | 4445 | 1536 | stop/stop | | [0260](items/0260.md) | correct | correct | 4200 | 2687 | stop/stop | | [0261](items/0261.md) | correct | correct | 2236 | 4618 | stop/stop | | [0262](items/0262.md) | correct | correct | 1754 | 1027 | stop/stop | | [0263](items/0263.md) | correct | correct | 2134 | 4457 | stop/stop | ## Unchanged wrong 41 questions. | Question | Base | Adapter | Base tokens | Adapter tokens | Finish (base/adapter) | |---|---|---|---:|---:|---| | [0064](items/0064.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0071](items/0071.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0082](items/0082.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0088](items/0088.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0094](items/0094.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0096](items/0096.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0100](items/0100.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0101](items/0101.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0103](items/0103.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0104](items/0104.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0108](items/0108.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0110](items/0110.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0118](items/0118.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0119](items/0119.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0120](items/0120.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0126](items/0126.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0128](items/0128.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0136](items/0136.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0138](items/0138.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0139](items/0139.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0141](items/0141.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0142](items/0142.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0145](items/0145.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0147](items/0147.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0151](items/0151.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0154](items/0154.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0161](items/0161.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0164](items/0164.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0166](items/0166.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0168](items/0168.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0173](items/0173.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0175](items/0175.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0176](items/0176.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0202](items/0202.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0204](items/0204.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0217](items/0217.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0219](items/0219.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0222](items/0222.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0223](items/0223.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0239](items/0239.md) | incorrect | incorrect | 32768 | 32768 | length/length | | [0240](items/0240.md) | incorrect | incorrect | 32768 | 32768 | length/length |