Spaces:

austindixson
/

python-sandbox-math-demo

Running

App Files Files Community

python-sandbox-math-demo / RECORDED.md

austindixson

Oct 30 full-context results; Oct 28 accuracy gains labeled as cap effects (#1)

5d18f7c 11 days ago

preview code

Raw History Blame Contribute Delete

4.74 kB

### Recorded results

The main win is speed: fewer generated tokens at about the same accuracy. The tool did not raise accuracy once the model had its full context. The AIME gains in the October 28 runs further down came from the token cap, because the no-tools side ran out of tokens more often.

### Full-context results (Oct 30, 2026)

Model: full-precision (BF16) LFM3-1.7B on a Hugging Face A100 with SGLang, 262,144-token context, the model card's sampling (temperature 1.0, top_p 0.95, top_k 20), one sample per problem, graded with `math_verify`. Up to 6 `run_python` calls, 10 s timeout. No answer ran out of context.

| Set | No tools | run_python | run_python + check |
|---|---|---|---|
| AIME 2025 (30) | 24/30 | 22/30 | 25/30 |
| AIME 2026 (30) | 26/30 | 25/30 | 25/30 |
| MATH-500 (500) | 486/500 | 486/500 | 481/500 |
| AIME tokens per solved problem | 21,776 | 16,688 (−23.4%) | 17,495 (−19.7%) |
| MATH-500 tokens per solved problem | 3,752 | 3,221 (−14.2%) | 3,510 (−6.4%) |

McNemar exact, two-sided: on all 60 AIME problems, run_python vs no tools fixed 3 and broke 6 (p = 0.51); run_python + check vs no tools fixed 5 and broke 5 (p = 1.0). On MATH-500 the p-values were 1.0 and 0.30. None of the accuracy differences is significant.

These cloud runs did not use the tool's network-denied sandbox. The Python code ran as the `nobody` user, and the network probe in each job's meta file returned `NET_OK`, so the code had network access there.

Write-up: https://www.destroyrebuild.xyz/blog/lfm3-python-tool-full-context

### Earlier capped runs (Oct 28, 2026)

These ran with a 25,000-token budget per problem. The accuracy changes below were cap effects and did not hold at full context. The token savings did.

| Set | Model | No tools | run_python | Tokens / problem | Change |
|---|---|---|---|---|---|
| MATH (30) | LFM3-1.7B | 30/30 | 29/30 | 3,036.8 → 2,022.8 | −33.4% |
| MATH (30) | Clover-35B-A3B | 28/30 | 28/30 | 2,791.7 → 1,458.6 | −47.8% |
| AIME 2025 (30) | LFM3-1.7B | 17/30 | 20/30 | 16,634.1 → 9,595.6 | −42.3% |
| MATH-500 L5 (60) | LFM3-1.7B | 54/60 | 50/60 | 7,577.1 → 5,971.6 | −21.2% |
| Combined (90) | LFM3-1.7B | 71/90 | 70/90 | 10,596.1 → 7,179.6 | −32.2% |
Time per problem on MATH (30): LFM3 24.18 s → 16.64 s (−31.2%). Clover 131.68 s → 69.44 s (−47.3%). I only quote tokens for AIME and L5, because part of the no-tools side ran with parallel workers on a shared server.
Accuracy (Wilson 95% CI, McNemar exact p):
- LFM3 on MATH: 30/30 (88.6–100%) without tools. 29/30 with Python by the grader, 30/30 by hand. One answer was a formatting miss.
- Clover on MATH: 28/30 both ways, p = 1.0.
- AIME 2025: 17/30 (39.2–72.6%) → 20/30 (48.8–80.8%), +10.0 pts. Fixed 4, broke 1, p = 0.375. Not significant. Token-cap hits 11 → 2.
- MATH-500 L5: 54/60 (79.9–95.3%) → 50/60 (72.0–90.7%), −6.7 pts. Fixed 0, broke 4, p = 0.125.
- Combined 90: fixed 4, broke 5, p = 1.0. Token-cap hits 14 → 3.
- `--verify` variant on AIME: 17/30 → 24/30 (62.7–90.5%), +23.3 pts, fixed 8, broke 1, p = 0.039. Tokens −29.8%. This is preliminary and post-hoc. I tried it after seeing the base results, and its L5 run had only 3 of 60 done at publish time.
Setup: llama.cpp `llama-server`, GGUF, greedy, thinking on, 25,000 generated-token budget per problem summed across all model calls. Up to 6 `run_python` calls, 10 s timeout per call, output capped at 2,000 characters. Apple M1 Max.

Raw data, the benchmark script and the grader: https://huggingface.co/austindixson/python-sandbox-math-tool

### Demo runs on 2 CPU threads (same app, 1,024-token cap)

Recorded before publishing, on a 2-thread Linux machine (bwrap sandbox) with this Space's own `live/app.py`. These are demo-setting numbers, not the eval above.

| Question | Correct answer | No tools | run_python |
|---|---|---|---|
| What is the sum of the digits of 2^1000? | 1366 | (no boxed answer) · 1,024 tok · 61.5 s | 1366 · 153 tok · 12.3 s |
| What is the remainder when 7^2026 is divided by 1000? | 649 | (no boxed answer) · 1,024 tok · 61.4 s | 649 · 281 tok · 19.8 s |
| What is 987654321 × 123456789? | 121932631112635269 | (no boxed answer) · 1,024 tok · 61.7 s | 121932631112635269 · 151 tok · 11.1 s |
| How many ways can you make change for one dollar using pennies, nickels, dimes and quarters? | 242 | (no boxed answer) · 1,024 tok · 59.4 s | 242 · 783 tok · 57.2 s |
| How many primes are there below 100,000? | 9592 | 9592 · 217 tok · 12.7 s | 9592 · 526 tok · 34.9 s |
A sixth question I tried (integers up to 10,000 divisible by 3 or 7 but not 5) hit the 1,024-token cap both ways without calling the tool, so I left it out of the presets.

	### Recorded results

	The main win is speed: fewer generated tokens at about the same accuracy. The tool did not raise accuracy once the model had its full context. The AIME gains in the October 28 runs further down came from the token cap, because the no-tools side ran out of tokens more often.

	### Full-context results (Oct 30, 2026)

	Model: full-precision (BF16) LFM3-1.7B on a Hugging Face A100 with SGLang, 262,144-token context, the model card's sampling (temperature 1.0, top_p 0.95, top_k 20), one sample per problem, graded with `math_verify`. Up to 6 `run_python` calls, 10 s timeout. No answer ran out of context.

	\| Set \| No tools \| run_python \| run_python + check \|
	\|---\|---\|---\|---\|
	\| AIME 2025 (30) \| 24/30 \| 22/30 \| 25/30 \|
	\| AIME 2026 (30) \| 26/30 \| 25/30 \| 25/30 \|
	\| MATH-500 (500) \| 486/500 \| 486/500 \| 481/500 \|
	\| AIME tokens per solved problem \| 21,776 \| 16,688 (−23.4%) \| 17,495 (−19.7%) \|
	\| MATH-500 tokens per solved problem \| 3,752 \| 3,221 (−14.2%) \| 3,510 (−6.4%) \|

	McNemar exact, two-sided: on all 60 AIME problems, run_python vs no tools fixed 3 and broke 6 (p = 0.51); run_python + check vs no tools fixed 5 and broke 5 (p = 1.0). On MATH-500 the p-values were 1.0 and 0.30. None of the accuracy differences is significant.

	These cloud runs did not use the tool's network-denied sandbox. The Python code ran as the `nobody` user, and the network probe in each job's meta file returned `NET_OK`, so the code had network access there.

	Write-up: https://www.destroyrebuild.xyz/blog/lfm3-python-tool-full-context

	### Earlier capped runs (Oct 28, 2026)

	These ran with a 25,000-token budget per problem. The accuracy changes below were cap effects and did not hold at full context. The token savings did.

	\| Set \| Model \| No tools \| run_python \| Tokens / problem \| Change \|
	\|---\|---\|---\|---\|---\|---\|
	\| MATH (30) \| LFM3-1.7B \| 30/30 \| 29/30 \| 3,036.8 → 2,022.8 \| −33.4% \|
	\| MATH (30) \| Clover-35B-A3B \| 28/30 \| 28/30 \| 2,791.7 → 1,458.6 \| −47.8% \|
	\| AIME 2025 (30) \| LFM3-1.7B \| 17/30 \| 20/30 \| 16,634.1 → 9,595.6 \| −42.3% \|
	\| MATH-500 L5 (60) \| LFM3-1.7B \| 54/60 \| 50/60 \| 7,577.1 → 5,971.6 \| −21.2% \|
	\| Combined (90) \| LFM3-1.7B \| 71/90 \| 70/90 \| 10,596.1 → 7,179.6 \| −32.2% \|

	Time per problem on MATH (30): LFM3 24.18 s → 16.64 s (−31.2%). Clover 131.68 s → 69.44 s (−47.3%). I only quote tokens for AIME and L5, because part of the no-tools side ran with parallel workers on a shared server.

	Accuracy (Wilson 95% CI, McNemar exact p):

	- LFM3 on MATH: 30/30 (88.6–100%) without tools. 29/30 with Python by the grader, 30/30 by hand. One answer was a formatting miss.
	- Clover on MATH: 28/30 both ways, p = 1.0.
	- AIME 2025: 17/30 (39.2–72.6%) → 20/30 (48.8–80.8%), +10.0 pts. Fixed 4, broke 1, p = 0.375. Not significant. Token-cap hits 11 → 2.
	- MATH-500 L5: 54/60 (79.9–95.3%) → 50/60 (72.0–90.7%), −6.7 pts. Fixed 0, broke 4, p = 0.125.
	- Combined 90: fixed 4, broke 5, p = 1.0. Token-cap hits 14 → 3.
	- `--verify` variant on AIME: 17/30 → 24/30 (62.7–90.5%), +23.3 pts, fixed 8, broke 1, p = 0.039. Tokens −29.8%. This is preliminary and post-hoc. I tried it after seeing the base results, and its L5 run had only 3 of 60 done at publish time.

	Setup: llama.cpp `llama-server`, GGUF, greedy, thinking on, 25,000 generated-token budget per problem summed across all model calls. Up to 6 `run_python` calls, 10 s timeout per call, output capped at 2,000 characters. Apple M1 Max.

	Raw data, the benchmark script and the grader: https://huggingface.co/austindixson/python-sandbox-math-tool

	### Demo runs on 2 CPU threads (same app, 1,024-token cap)

	Recorded before publishing, on a 2-thread Linux machine (bwrap sandbox) with this Space's own `live/app.py`. These are demo-setting numbers, not the eval above.

	\| Question \| Correct answer \| No tools \| run_python \|
	\|---\|---\|---\|---\|
	\| What is the sum of the digits of 2^1000? \| 1366 \| (no boxed answer) · 1,024 tok · 61.5 s \| 1366 · 153 tok · 12.3 s \|
	\| What is the remainder when 7^2026 is divided by 1000? \| 649 \| (no boxed answer) · 1,024 tok · 61.4 s \| 649 · 281 tok · 19.8 s \|
	\| What is 987654321 × 123456789? \| 121932631112635269 \| (no boxed answer) · 1,024 tok · 61.7 s \| 121932631112635269 · 151 tok · 11.1 s \|
	\| How many ways can you make change for one dollar using pennies, nickels, dimes and quarters? \| 242 \| (no boxed answer) · 1,024 tok · 59.4 s \| 242 · 783 tok · 57.2 s \|
	\| How many primes are there below 100,000? \| 9592 \| 9592 · 217 tok · 12.7 s \| 9592 · 526 tok · 34.9 s \|

	A sixth question I tried (integers up to 10,000 divisible by 3 or 7 but not 5) hit the 1,024-token cap both ways without calling the tool, so I left it out of the presets.