Instructions to use hunterbown/trainless-math-2b-loop-repair with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use hunterbown/trainless-math-2b-loop-repair with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-2B") model = PeftModel.from_pretrained(base_model, "hunterbown/trainless-math-2b-loop-repair") - Notebooks
- Google Colab
- Kaggle
Download evaluation/README.md from hunterbown/trainless-math-2b-loop-repair: direct link, hf CLI and curl.
- Browser
- Download file 17.5 kB
-
https://huggingface.co/hunterbown/trainless-math-2b-loop-repair/resolve/main/evaluation/README.md
- Command line
-
hf download hf://hunterbown/trainless-math-2b-loop-repair/evaluation/README.md
-
curl -L -o README.md https://huggingface.co/hunterbown/trainless-math-2b-loop-repair/resolve/main/evaluation/README.md
Full 200-question evaluation
Every question, reference answer and complete response is included here, including all seven regressions. This is the existing MATH-500 slice 64–263, not a new held-out discovery test.
| Arm | Correct | Wilson 95% interval | Capped at 32,768 tokens |
|---|---|---|---|
| Base | 136/200 (68.0%) | 61.2–74.1% | 63/200 |
| Adapter | 152/200 (76.0%) | 69.6–81.4% | 48/200 |
The adapter fixes 23 answers and loses 7: a paired gain of 8 percentage points, bootstrap 95% interval 3–13.5 points. Both arms took about 760 seconds; this run establishes no wall-clock speedup.
Fixed · Lost · Unchanged correct · Unchanged wrong
Downloadable data
- questions.jsonl: all 200 questions, reference answers, IDs/order and full reconstructed input strings.
- responses.jsonl.gz: both full outputs and original correct/finish/token flags, with no truncation.
- summary.csv: spreadsheet-safe overview; potentially executable text cells are prefixed with an apostrophe. JSONL retains exact original strings.
- protocol.json: full settings, runtime, model/data/source hashes and scope.
- Attribution and dataset license.
- File manifest.
Original generation JSON did not save input strings. The complete inputs here were reconstructed with the pinned base tokenizer and the same historical Transformers version, thinking enabled. The old Hub dataset revision was not saved; separately verified current upstream correspondence is labeled as such. Native LaTeX and all untrusted question/response text are preserved literally inside safe fences in review pages. Stored automatic labels are not a guarantee that every claim in an answer is correct.
Fixed
23 questions.
| Question | Base | Adapter | Base tokens | Adapter tokens | Finish (base/adapter) |
|---|---|---|---|---|---|
| 0076 | incorrect | correct | 32768 | 12929 | length/stop |
| 0080 | incorrect | correct | 32768 | 21020 | length/stop |
| 0099 | incorrect | correct | 32768 | 15684 | length/stop |
| 0123 | incorrect | correct | 32768 | 29769 | length/stop |
| 0130 | incorrect | correct | 32768 | 9066 | length/stop |
| 0150 | incorrect | correct | 32768 | 27507 | length/stop |
| 0152 | incorrect | correct | 32768 | 30347 | length/stop |
| 0153 | incorrect | correct | 32768 | 4777 | length/stop |
| 0156 | incorrect | correct | 32768 | 27104 | length/stop |
| 0157 | incorrect | correct | 32768 | 7016 | length/stop |
| 0177 | incorrect | correct | 32768 | 27852 | length/stop |
| 0181 | incorrect | correct | 32768 | 8458 | length/stop |
| 0194 | incorrect | correct | 32768 | 25370 | length/stop |
| 0206 | incorrect | correct | 32768 | 5250 | length/stop |
| 0210 | incorrect | correct | 32768 | 21893 | length/stop |
| 0213 | incorrect | correct | 32768 | 31622 | length/stop |
| 0225 | incorrect | correct | 20543 | 20463 | stop/stop |
| 0226 | incorrect | correct | 32768 | 7855 | length/stop |
| 0228 | incorrect | correct | 32768 | 30724 | length/stop |
| 0232 | incorrect | correct | 32768 | 26377 | length/stop |
| 0242 | incorrect | correct | 32768 | 5985 | length/stop |
| 0246 | incorrect | correct | 32768 | 14914 | length/stop |
| 0254 | incorrect | correct | 32768 | 13506 | length/stop |
Lost
7 questions.
| Question | Base | Adapter | Base tokens | Adapter tokens | Finish (base/adapter) |
|---|---|---|---|---|---|
| 0091 | correct | incorrect | 6669 | 32768 | stop/length |
| 0140 | correct | incorrect | 16502 | 32768 | stop/length |
| 0170 | correct | incorrect | 31784 | 32768 | stop/length |
| 0188 | correct | incorrect | 29414 | 32768 | stop/length |
| 0200 | correct | incorrect | 2776 | 32768 | stop/length |
| 0236 | correct | incorrect | 27500 | 32768 | stop/length |
| 0245 | correct | incorrect | 22065 | 32768 | stop/length |
Unchanged correct
129 questions.
| Question | Base | Adapter | Base tokens | Adapter tokens | Finish (base/adapter) |
|---|---|---|---|---|---|
| 0065 | correct | correct | 6782 | 1708 | stop/stop |
| 0066 | correct | correct | 7999 | 3454 | stop/stop |
| 0067 | correct | correct | 3234 | 2946 | stop/stop |
| 0068 | correct | correct | 21276 | 16540 | stop/stop |
| 0069 | correct | correct | 5064 | 5327 | stop/stop |
| 0070 | correct | correct | 6903 | 7178 | stop/stop |
| 0072 | correct | correct | 6780 | 16977 | stop/stop |
| 0073 | correct | correct | 5842 | 7857 | stop/stop |
| 0074 | correct | correct | 5142 | 3073 | stop/stop |
| 0075 | correct | correct | 9704 | 2920 | stop/stop |
| 0077 | correct | correct | 1678 | 1040 | stop/stop |
| 0078 | correct | correct | 22911 | 10631 | stop/stop |
| 0079 | correct | correct | 1829 | 1613 | stop/stop |
| 0081 | correct | correct | 16315 | 2055 | stop/stop |
| 0083 | correct | correct | 30575 | 15493 | stop/stop |
| 0084 | correct | correct | 10286 | 3127 | stop/stop |
| 0085 | correct | correct | 7172 | 1993 | stop/stop |
| 0086 | correct | correct | 1233 | 792 | stop/stop |
| 0087 | correct | correct | 2759 | 3093 | stop/stop |
| 0089 | correct | correct | 13973 | 10862 | stop/stop |
| 0090 | correct | correct | 6740 | 4882 | stop/stop |
| 0092 | correct | correct | 7157 | 7153 | stop/stop |
| 0093 | correct | correct | 3919 | 3556 | stop/stop |
| 0095 | correct | correct | 8832 | 7284 | stop/stop |
| 0097 | correct | correct | 11092 | 2520 | stop/stop |
| 0098 | correct | correct | 4948 | 2534 | stop/stop |
| 0102 | correct | correct | 3761 | 1639 | stop/stop |
| 0105 | correct | correct | 2743 | 1706 | stop/stop |
| 0106 | correct | correct | 3434 | 2219 | stop/stop |
| 0107 | correct | correct | 3285 | 2336 | stop/stop |
| 0109 | correct | correct | 12772 | 13784 | stop/stop |
| 0111 | correct | correct | 1545 | 1894 | stop/stop |
| 0112 | correct | correct | 14023 | 13432 | stop/stop |
| 0113 | correct | correct | 5916 | 5128 | stop/stop |
| 0114 | correct | correct | 3941 | 8776 | stop/stop |
| 0115 | correct | correct | 2682 | 8385 | stop/stop |
| 0116 | correct | correct | 5047 | 4310 | stop/stop |
| 0117 | correct | correct | 1035 | 1457 | stop/stop |
| 0121 | correct | correct | 4496 | 1586 | stop/stop |
| 0122 | correct | correct | 3371 | 1815 | stop/stop |
| 0124 | correct | correct | 7955 | 4731 | stop/stop |
| 0125 | correct | correct | 2537 | 5048 | stop/stop |
| 0127 | correct | correct | 4864 | 3671 | stop/stop |
| 0129 | correct | correct | 21718 | 10436 | stop/stop |
| 0131 | correct | correct | 9718 | 7151 | stop/stop |
| 0132 | correct | correct | 4296 | 5744 | stop/stop |
| 0133 | correct | correct | 31709 | 15540 | stop/stop |
| 0134 | correct | correct | 11333 | 12661 | stop/stop |
| 0135 | correct | correct | 3578 | 6782 | stop/stop |
| 0137 | correct | correct | 6953 | 18823 | stop/stop |
| 0143 | correct | correct | 4296 | 5461 | stop/stop |
| 0144 | correct | correct | 1175 | 982 | stop/stop |
| 0146 | correct | correct | 1714 | 2284 | stop/stop |
| 0148 | correct | correct | 1395 | 1776 | stop/stop |
| 0149 | correct | correct | 8739 | 3726 | stop/stop |
| 0155 | correct | correct | 11464 | 4024 | stop/stop |
| 0158 | correct | correct | 6105 | 4909 | stop/stop |
| 0159 | correct | correct | 7537 | 2040 | stop/stop |
| 0160 | correct | correct | 4770 | 3190 | stop/stop |
| 0162 | correct | correct | 11483 | 11391 | stop/stop |
| 0163 | correct | correct | 15302 | 7822 | stop/stop |
| 0165 | correct | correct | 2612 | 1374 | stop/stop |
| 0167 | correct | correct | 8614 | 866 | stop/stop |
| 0169 | correct | correct | 3153 | 2411 | stop/stop |
| 0171 | correct | correct | 1605 | 940 | stop/stop |
| 0172 | correct | correct | 2125 | 2315 | stop/stop |
| 0174 | correct | correct | 4310 | 1693 | stop/stop |
| 0178 | correct | correct | 5596 | 3860 | stop/stop |
| 0179 | correct | correct | 9616 | 7404 | stop/stop |
| 0180 | correct | correct | 14213 | 9351 | stop/stop |
| 0182 | correct | correct | 1574 | 1225 | stop/stop |
| 0183 | correct | correct | 18030 | 11485 | stop/stop |
| 0184 | correct | correct | 32337 | 26180 | stop/stop |
| 0185 | correct | correct | 1203 | 2285 | stop/stop |
| 0186 | correct | correct | 9637 | 20581 | stop/stop |
| 0187 | correct | correct | 4253 | 2048 | stop/stop |
| 0189 | correct | correct | 25279 | 21996 | stop/stop |
| 0190 | correct | correct | 4365 | 2866 | stop/stop |
| 0191 | correct | correct | 1298 | 988 | stop/stop |
| 0192 | correct | correct | 6075 | 6742 | stop/stop |
| 0193 | correct | correct | 24085 | 19913 | stop/stop |
| 0195 | correct | correct | 11904 | 10076 | stop/stop |
| 0196 | correct | correct | 4758 | 2756 | stop/stop |
| 0197 | correct | correct | 16517 | 14823 | stop/stop |
| 0198 | correct | correct | 3640 | 14661 | stop/stop |
| 0199 | correct | correct | 2926 | 1362 | stop/stop |
| 0201 | correct | correct | 10306 | 9841 | stop/stop |
| 0203 | correct | correct | 6301 | 5588 | stop/stop |
| 0205 | correct | correct | 8882 | 10808 | stop/stop |
| 0207 | correct | correct | 1248 | 1633 | stop/stop |
| 0208 | correct | correct | 5032 | 3356 | stop/stop |
| 0209 | correct | correct | 6931 | 3565 | stop/stop |
| 0211 | correct | correct | 6316 | 2495 | stop/stop |
| 0212 | correct | correct | 2776 | 4866 | stop/stop |
| 0214 | correct | correct | 13527 | 16534 | stop/stop |
| 0215 | correct | correct | 5715 | 6024 | stop/stop |
| 0216 | correct | correct | 2609 | 5307 | stop/stop |
| 0218 | correct | correct | 901 | 827 | stop/stop |
| 0220 | correct | correct | 12714 | 10089 | stop/stop |
| 0221 | correct | correct | 17409 | 30557 | stop/stop |
| 0224 | correct | correct | 18795 | 15996 | stop/stop |
| 0227 | correct | correct | 11167 | 3798 | stop/stop |
| 0229 | correct | correct | 2621 | 1368 | stop/stop |
| 0230 | correct | correct | 22055 | 16334 | stop/stop |
| 0231 | correct | correct | 4955 | 6047 | stop/stop |
| 0233 | correct | correct | 3999 | 8428 | stop/stop |
| 0234 | correct | correct | 19969 | 10600 | stop/stop |
| 0235 | correct | correct | 3012 | 1843 | stop/stop |
| 0237 | correct | correct | 7181 | 4019 | stop/stop |
| 0238 | correct | correct | 10822 | 13250 | stop/stop |
| 0241 | correct | correct | 11076 | 14645 | stop/stop |
| 0243 | correct | correct | 7571 | 2695 | stop/stop |
| 0244 | correct | correct | 8781 | 9156 | stop/stop |
| 0247 | correct | correct | 8048 | 4432 | stop/stop |
| 0248 | correct | correct | 21429 | 21631 | stop/stop |
| 0249 | correct | correct | 14585 | 15530 | stop/stop |
| 0250 | correct | correct | 2719 | 4719 | stop/stop |
| 0251 | correct | correct | 17797 | 17509 | stop/stop |
| 0252 | correct | correct | 6486 | 3271 | stop/stop |
| 0253 | correct | correct | 2956 | 2425 | stop/stop |
| 0255 | correct | correct | 5800 | 3611 | stop/stop |
| 0256 | correct | correct | 7299 | 3105 | stop/stop |
| 0257 | correct | correct | 4773 | 2002 | stop/stop |
| 0258 | correct | correct | 5517 | 9361 | stop/stop |
| 0259 | correct | correct | 4445 | 1536 | stop/stop |
| 0260 | correct | correct | 4200 | 2687 | stop/stop |
| 0261 | correct | correct | 2236 | 4618 | stop/stop |
| 0262 | correct | correct | 1754 | 1027 | stop/stop |
| 0263 | correct | correct | 2134 | 4457 | stop/stop |
Unchanged wrong
41 questions.
| Question | Base | Adapter | Base tokens | Adapter tokens | Finish (base/adapter) |
|---|---|---|---|---|---|
| 0064 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0071 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0082 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0088 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0094 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0096 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0100 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0101 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0103 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0104 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0108 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0110 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0118 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0119 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0120 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0126 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0128 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0136 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0138 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0139 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0141 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0142 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0145 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0147 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0151 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0154 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0161 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0164 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0166 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0168 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0173 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0175 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0176 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0202 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0204 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0217 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0219 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0222 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0223 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0239 | incorrect | incorrect | 32768 | 32768 | length/length |
| 0240 | incorrect | incorrect | 32768 | 32768 | length/length |