Static replay of recorded with/without run_python runs

README.md

@@ -1,10 +1,25 @@
 ---
-title: Python Sandbox Math Demo
-emoji: 📈
+title: Python sandbox math demo
+emoji: 🧮
 colorFrom: blue
-colorTo: pink
+colorTo: gray
 sdk: static
+app_file: index.html
 pinned: false
+license: mit
+short_description: Same question with and without a run_python tool.
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Python sandbox math demo
+
+This Space shows a small local model (LFM3-1.7B) answering the same math question twice: once on its own, and once with a sandboxed `run_python` tool. For each side you see the answer, the generated tokens, the time and the tool calls.
+
+**It replays recorded runs. It is not live.** Hugging Face now needs a PRO subscription to host a Gradio Space on free CPU, so this is a static Space. The runs were recorded with `app.py` (included here) on a 2-thread Linux machine using the bwrap sandbox backend.
+
+Tool code and results: https://huggingface.co/austindixson/python-sandbox-math-tool
+
+Notes:
+
+- To run it live yourself: `pip install -r requirements.txt && python app.py`. It downloads llama.cpp b11938 and the LFM3-1.7B Q4_0 GGUF on first start (Linux x64). On 2 CPU cores, expect up to a minute per side.
+- The demo caps each answer at 1,024 tokens. That is not the 25,000-token budget used in the recorded evaluation. A small cap exaggerates the gap, because the no-tools side often runs out before it finishes.
+- On Linux the sandbox is bubblewrap if user namespaces work, otherwise a weaker fallback. On a Mac the tool uses `sandbox-exec`, which is stronger.

RECORDED.md

@@ -0,0 +1,40 @@
+### Recorded results
+
+The main win is fewer generated tokens and less time at about the same accuracy. Accuracy rose on AIME 2025 but not significantly for the base prompt. On MATH-500 level 5 it went down slightly.
+
+| Set | Model | No tools | run_python | Tokens / problem | Change |
+|---|---|---|---|---|---|
+| MATH (30) | LFM3-1.7B | 30/30 | 29/30 | 3,036.8 → 2,022.8 | −33.4% |
+| MATH (30) | Clover-35B-A3B | 28/30 | 28/30 | 2,791.7 → 1,458.6 | −47.8% |
+| AIME 2025 (30) | LFM3-1.7B | 17/30 | 20/30 | 16,634.1 → 9,595.6 | −42.3% |
+| MATH-500 L5 (60) | LFM3-1.7B | 54/60 | 50/60 | 7,577.1 → 5,971.6 | −21.2% |
+| Combined (90) | LFM3-1.7B | 71/90 | 70/90 | 10,596.1 → 7,179.6 | −32.2% |
+
+Time per problem on MATH (30): LFM3 24.18 s → 16.64 s (−31.2%). Clover 131.68 s → 69.44 s (−47.3%). I only quote tokens for AIME and L5, because part of the no-tools side ran with parallel workers on a shared server.
+
+Accuracy (Wilson 95% CI, McNemar exact p):
+
+- LFM3 on MATH: 30/30 (88.6–100%) without tools. 29/30 with Python by the grader, 30/30 by hand. One answer was a formatting miss.
+- Clover on MATH: 28/30 both ways, p = 1.0.
+- AIME 2025: 17/30 (39.2–72.6%) → 20/30 (48.8–80.8%), +10.0 pts. Fixed 4, broke 1, p = 0.375. Not significant. Token-cap hits 11 → 2.
+- MATH-500 L5: 54/60 (79.9–95.3%) → 50/60 (72.0–90.7%), −6.7 pts. Fixed 0, broke 4, p = 0.125.
+- Combined 90: fixed 4, broke 5, p = 1.0. Token-cap hits 14 → 3.
+- `--verify` variant on AIME: 17/30 → 24/30 (62.7–90.5%), +23.3 pts, fixed 8, broke 1, p = 0.039. Tokens −29.8%. This is preliminary and post-hoc. I tried it after seeing the base results, and its L5 run had only 3 of 60 done at publish time.
+
+Setup: llama.cpp `llama-server`, GGUF, greedy, thinking on, 25,000 generated-token budget per problem summed across all model calls. Up to 6 `run_python` calls, 10 s timeout per call, output capped at 2,000 characters. Apple M1 Max.
+
+Raw data, the benchmark script and the grader: https://huggingface.co/austindixson/python-sandbox-math-tool
+
+### Demo runs on 2 CPU threads (same app, 1,024-token cap)
+
+Recorded before publishing, on a 2-thread Linux machine (bwrap sandbox) with this Space's own `app.py`. These are demo-setting numbers, not the eval above.
+
+| Question | Correct answer | No tools | run_python |
+|---|---|---|---|
+| What is the sum of the digits of 2^1000? | 1366 | (no boxed answer) · 1,024 tok · 61.5 s | 1366 · 153 tok · 12.3 s |
+| What is the remainder when 7^2026 is divided by 1000? | 649 | (no boxed answer) · 1,024 tok · 61.4 s | 649 · 281 tok · 19.8 s |
+| What is 987654321 × 123456789? | 121932631112635269 | (no boxed answer) · 1,024 tok · 61.7 s | 121932631112635269 · 151 tok · 11.1 s |
+| How many ways can you make change for one dollar using pennies, nickels, dimes and quarters? | 242 | (no boxed answer) · 1,024 tok · 59.4 s | 242 · 783 tok · 57.2 s |
+| How many primes are there below 100,000? | 9592 | 9592 · 217 tok · 12.7 s | 9592 · 526 tok · 34.9 s |
+
+A sixth question I tried (integers up to 10,000 divisible by 3 or 7 but not 5) hit the 1,024-token cap both ways without calling the tool, so I left it out of the presets.

index.html

@@ -1,19 +1,42 @@
-<!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
-</html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Python sandbox math demo: with vs without a run_python tool</title>
+<script src="https://cdn.jsdelivr.net/npm/marked@12/marked.min.js"></script>
+<style>
+body{font-family:system-ui,-apple-system,Segoe UI,sans-serif;max-width:1150px;margin:0 auto;padding:20px;color:#1f2328;line-height:1.5}
+h1{font-size:1.6em;margin-bottom:.2em} .note{background:#fff8e6;border:1px solid #f0d58a;padding:10px 14px;border-radius:8px;margin:12px 0}
+.tabs button{font-size:1em;padding:8px 14px;border:1px solid #ccc;background:#f6f8fa;border-radius:8px 8px 0 0;cursor:pointer}
+.tabs button.on{background:#fff;border-bottom-color:#fff;font-weight:600}
+.panel{border:1px solid #ccc;padding:16px;border-radius:0 8px 8px 8px}
+select{font-size:1em;padding:6px;max-width:100%} .cols{display:grid;grid-template-columns:1fr 1fr;gap:16px;margin-top:12px}
+@media(max-width:800px){.cols{grid-template-columns:1fr}}
+.side{border:1px solid #ddd;border-radius:8px;padding:12px;background:#fafbfc}
+pre{white-space:pre-wrap;word-break:break-word;background:#f3f4f6;padding:8px;border-radius:6px;font-size:.88em;max-height:420px;overflow:auto}
+table{border-collapse:collapse;font-size:.92em} td,th{border:1px solid #ddd;padding:4px 8px} img{max-width:100%}
+</style></head><body>
+<h1>Small model, with and without a Python tool</h1>
+<p>The same model (LiquidAI <b>LFM3-1.7B</b>, QAD-Q4_0 GGUF, llama.cpp b11938, greedy, thinking off) answers a math question twice: once on its own, and once with a sandboxed <code>run_python</code> tool it can call up to 6 times.
+Tool code, benchmark and raw results: <a href="https://huggingface.co/austindixson/python-sandbox-math-tool">austindixson/python-sandbox-math-tool</a>.</p>
+<div class="note"><b>This page replays recorded runs. It is not live.</b> Hugging Face now needs a PRO subscription to host a Gradio Space on the free CPU tier, so this Space is static.
+The runs below were recorded with this Space's own <code>app.py</code> on a 2-thread Linux machine (sandbox backend: <code>bwrap</code>), with a 1,024-token cap per answer.
+That cap is a demo setting. It is not the 25,000-token budget used in the evaluation, and a small cap exaggerates the gap because the no-tools side often runs out before it answers.
+To run it live, use <code>app.py</code> in this repo (<code>pip install -r requirements.txt && python app.py</code>) or the proxy in the tool repo. On a Mac the tool uses <code>sandbox-exec</code> (network denied by the OS, writes only in a temp dir). On Linux it uses bubblewrap when user namespaces work, and a weaker fallback otherwise.</div>
+<div class="tabs"><button id="t1" class="on" onclick="tab(1)">Side by side (recorded)</button><button id="t2" onclick="tab(2)">Recorded results</button></div>
+<div class="panel" id="p1">
+<label>Question: <select id="q" onchange="show()"></select></label>
+<div class="cols"><div class="side"><h3>No tools</h3><div id="ns"></div><details open><summary>Model output</summary><pre id="nt"></pre></details></div>
+<div class="side"><h3>With run_python</h3><div id="ps"></div><details open><summary>Model output</summary><pre id="pt"></pre></details><details open><summary>Tool calls and outputs</summary><div id="pc"></div></details></div></div>
+</div>
+<div class="panel" id="p2" style="display:none"><div id="rec"></div>
+<h3>Tokens per problem</h3><img src="fig_tokens.png" alt="tokens per problem chart"><h3>AIME 2025 accuracy</h3><img src="fig_aime.png" alt="AIME accuracy chart"></div>
+<script>
+const RUNS=[{"q": "What is the sum of the digits of 2^1000?", "gold": "1366", "none": ["**Final answer:** `(no boxed answer)`  \n**Generated tokens:** 1,024  \u00b7  **Time:** 61.5 s  \u00b7  **Model calls:** 1  \u00b7  **Tool calls:** 0  \n\u26a0\ufe0f hit the token cap", "To find the sum of the digits of \\(2^{1000}\\), we need to determine the decimal representation of \\(2^{1000}\\) and then add its digits. However, calculating \\(2^{1000}\\) directly is impractical by hand due to its large size (it has 302 digits). Instead, we can use properties of modular arithmetic or known results about the digit sum.\n\nThe sum of the digits of a number is congruent to the number modulo 9. This is because the digital root (and thus the sum of digits) is related to \\(n \\mod 9\\). Specifically:\n\\[\n\\text{Sum of digits} \\equiv n \\pmod{9}\n\\]\nBut this only gives us the value modulo 9, not the actual sum. For large exponents like \\(2^{1000}\\), we can compute \\(2^{1000} \\mod 9\\) to find a clue about the digit sum.\n\nFirst, note that:\n\\[\n2^1 = 2 \\equiv 2 \\pmod{9}\n\\]\n\\[\n2^2 = 4 \\equiv 4 \\pmod{9}\n\\]\n\\[\n2^3 = 8 \\equiv 8 \\pmod{9}\n\\]\n\\[\n2^4 = 16 \\equiv 7 \\pmod{9} \\quad (\\text{since } 16 - 9 = 7)\n\\]\n\\[\n2^5 = 32 \\equiv 5 \\pmod{9} \\quad (32 - 27 = 5)\n\\]\n\\[\n2^6 = 64 \\equiv 1 \\pmod{9} \\quad (64 - 63 = 1)\n\\]\nThe powers of 2 modulo 9 cycle every 6: \\(2, 4, 8, 7, 5, 1\\). Since \\(1000 \\div 6 = 166\\) remainder \\(4\\), we have:\n\\[\n2^{1000} \\equiv 2^4 \\equiv 16 \\equiv 7 \\pmod{9}\n\\]\nSo, the sum of the digits of \\(2^{1000}\\) is congruent to 7 modulo 9. That is, if \\(S\\) is the sum of the digits, then:\n\\[\nS \\equiv 7 \\pmod{9}\n\\]\n\nNow, we need the actual value or a way to compute it. There are known results for the digit sums of powers of 2. Alternatively, we can use computational tools or properties. Since this is a text-based reasoning, I recall that \\(2^{1000}\\) has been computed and its digits summed before. However, let's verify using another approach.\n\nWe can also compute the number of digits: \n\\[\n\\log_{10}(2^{1000}) = 1000 \\log_{10} 2 \\approx 1000 \\times 0.30102999566 = 301.02999566\n\\]\nSo \\(2^{1000}\\) has \\(\\lfloor 301.02999566 \\rfloor + 1 = 302\\) digits.\n\nTo find the digit sum, one might need to look up or compute it. Since this is a known problem, I remember that the sum of the digits of \\(2^{1000}\\) is 1367? But let's not guess; we can reason differently.\n\nAlternatively, there is a method using programming or known sequences. For the purpose of this problem, since it's likely expecting an answer from standard knowledge or calculation, and given the modulo 9 result is 7, but that doesn't give the sum directly.\n\nWait, perhaps I can find \\(2^{1000}\\) mod 99 to get more info? But digit sum isn't directly related to mod 99. Another idea: use the fact that for large numbers, the digit sum can be found via software. Since this is a text-based interaction and I need to provide an answer, I'll rely on known results or compute step by step.\n\nUpon recalling or using computational knowledge (as in typical contest problems), the sum of digits of \\(2^{1000}\\) is 1367? Let me check consistency: if S \u2261 7 mod 9, then 1367 \u00f7 9 = 151 * 9 = 1359, remainder 8. That's not 7. So that's wrong.\n\nLet me compute \\(2^{10} = 1024\\), sum of digits = 1+0+2+4=7.\n\\(2^{20}\\): \\(2^{10}^2 = 1024^2 = 1048576\\), sum: 1+0+4+8+5+7+6=31? Wait, better to use known values.\n\nActually, I can compute the digit sum for smaller powers and see a pattern, but it's not straightforward. Given time constraints, I'll assume that the correct answer is known or derived from standard sources. \n\nAlternatively, we can use the fact that \\(2^{1000} = (2^{10})^{100}\\). But still large.\n\nI found online resources or", "_no tool calls_"], "python": ["**Final answer:** `1366`  \n**Generated tokens:** 153  \u00b7  **Time:** 12.3 s  \u00b7  **Model calls:** 2  \u00b7  **Tool calls:** 1", "The sum of the digits of \\(2^{1000}\\) is \\(\\boxed{1366}\\).", "**run_python #1**\n```python\nn = 2**1000\ns = str(n)\ndigit_sum = sum(int(ch) for ch in s)\nprint(digit_sum)\n```\n```text\n1366\n```"]}, {"q": "What is the remainder when 7^2026 is divided by 1000?", "gold": "649", "none": ["**Final answer:** `(no boxed answer)`  \n**Generated tokens:** 1,024  \u00b7  **Time:** 61.4 s  \u00b7  **Model calls:** 1  \u00b7  **Tool calls:** 0  \n\u26a0\ufe0f hit the token cap", "To find the remainder when \\(7^{2026}\\) is divided by 1000, we need to compute \\(7^{2026} \\mod 1000\\). Since \\(1000 = 8 \\times 125\\) and 8 and 125 are coprime, we can use the Chinese Remainder Theorem (CRT) by finding the remainders modulo 8 and modulo 125 separately.\n\n**Step 1: Compute \\(7^{2026} \\mod 8\\)**\n\nNote that:\n- \\(7 \\equiv -1 \\pmod{8}\\)\n- So, \\(7^{2026} \\equiv (-1)^{2026} \\pmod{8}\\)\n\nSince 2026 is even, \\((-1)^{2026} = 1\\). Thus,\n\\[\n7^{2026} \\equiv 1 \\pmod{8}\n\\]\n\n**Step 2: Compute \\(7^{2026} \\mod 125\\)**\n\nWe need to find the exponent modulo \\(\\phi(125)\\), where \\(\\phi\\) is Euler's totient function. Since 125 is a power of an odd prime, \\(\\phi(125) = 125 \\times (1 - \\frac{1}{5}) = 100\\).\n\nThus,\n\\[\n7^{2026} \\equiv 7^{2026 \\mod 100} \\pmod{125}\n\\]\nCalculate \\(2026 \\div 100\\):\n- \\(100 \\times 20 = 2000\\), so \\(2026 - 2000 = 26\\). Thus, \\(2026 \\equiv 26 \\pmod{100}\\).\n\nSo,\n\\[\n7^{2026} \\equiv 7^{26} \\pmod{125}\n\\]\n\nNow compute \\(7^{26} \\mod 125\\) using successive squaring:\n\n- \\(7^1 = 7\\)\n- \\(7^2 = 49\\)\n- \\(7^4 = (7^2)^2 = 49^2 = 2401 \\equiv 2401 - 19 \\times 125 = 2401 - 2375 = 26 \\pmod{125}\\)\n- \\(7^8 = (7^4)^2 = 26^2 = 676 \\equiv 676 - 5 \\times 125 = 676 - 625 = 51 \\pmod{125}\\)\n- \\(7^{16} = (7^8)^2 = 51^2 = 2601 \\equiv 2601 - 20 \\times 125 = 2601 - 2500 = 101 \\pmod{125}\\)\n\nNow, express \\(26\\) in binary: \\(26 = 16 + 8 + 2\\). Thus,\n\\[\n7^{26} = 7^{16} \\times 7^8 \\times 7^2 \\equiv 101 \\times 51 \\times 49 \\pmod{125}\n\\]\n\nFirst, compute \\(101 \\times 51\\):\n- \\(100 \\times 51 = 5100\\)\n- \\(1 \\times 51 = 51\\)\n- Total: \\(5100 + 51 = 5151\\)\n\nNow reduce modulo 125:\n- \\(125 \\times 41 = 5125\\)\n- \\(5151 - 5125 = 26\\), so \\(101 \\times 51 \\equiv 26 \\pmod{125}\\)\n\nNext, multiply by 49:\n\\[\n26 \\times 49 = 1274\n\\]\nReduce modulo 125:\n- \\(125 \\times 10 = 1250\\)\n- \\(1274 - 1250 = 24\\), so \\(26 \\times 49 \\equiv 24 \\pmod{125}\\)\n\nThus,\n\\[\n7^{26} \\equiv 24 \\pmod{125}\n\\]\n\nSo, we have:\n\\[\n7^{2026} \\equiv 1 \\pmod{8} \\quad \\text{and} \\quad 7^{2026} \\equiv 24 \\pmod{125}\n\\]\n\n**Step 3: Apply Chinese Remainder Theorem**\n\nWe need to find a number \\(x\\) such that:\n\\[\n\\begin{cases}\nx \\equiv 1 \\pmod{8} \\\\\nx \\equiv 24 \\pmod{125}\n\\end{cases}\n\\]\n\nLet \\(x = 125k + 24\\) for some integer \\(k\\). Substitute into the first congruence:\n\\[\n125k + 24 \\equiv 1 \\pmod{8}\n\\]\nCompute \\(125 \\mod 8\\):\n- \\(125 \\div 8 = 15 \\times 8 = 120\\), remainder 5, so \\(125 \\equiv 5 \\pmod{8}\\)\nThus,\n\\[\n5k + 24 \\equiv 1 \\pmod{8} \\implies 5k + 0 \\equiv 1 \\pmod{8} \\quad (\\text{since } 24 \\div 8 = 3)\n\\]\nSo,\n\\[\n5k \\equiv 1 \\pmod{8}\n\\]\n\nSolve for \\(k\\): Multiply both sides by the modular inverse of 5 modulo 8. Since \\(5 \\times", "_no tool calls_"], "python": ["**Final answer:** `649`  \n**Generated tokens:** 281  \u00b7  **Time:** 19.8 s  \u00b7  **Model calls:** 2  \u00b7  **Tool calls:** 1", "We need the remainder when \\(7^{2026}\\) is divided by 1000. Computing directly with modular exponentiation gives:\n\n\\[\n7^{2026} \\bmod 1000 = 649.\n\\]\n\nThus, the remainder is \\(\\boxed{649}\\).", "**run_python #1**\n```python\nprint(pow(7, 2026, 1000))\n\n```\n```text\n649\n```"]}, {"q": "What is 987654321 \u00d7 123456789?", "gold": "121932631112635269", "none": ["**Final answer:** `(no boxed answer)`  \n**Generated tokens:** 1,024  \u00b7  **Time:** 61.7 s  \u00b7  **Model calls:** 1  \u00b7  **Tool calls:** 0  \n\u26a0\ufe0f hit the token cap", "We need to compute the product \\(987,654,321 \\times 123,456,789\\). \n\nNotice that these numbers are related: \n\\(987,654,321 = 10^9 - 12,345,679\\)? Not exactly. Alternatively, observe that:\n\\[\n987,654,321 = 1,000,000,000 - 12,345,679\n\\]\nBut maybe there's a pattern with the digits. Another approach: \nLet \\(a = 123,456,789\\). Then note that \\(987,654,321\\) is like reversing the digits of \\(123,456,789\\)? Actually, \\(123,456,789\\) reversed is \\(987,654,321\\). Yes! Because:\n- Original: 1 2 3 4 5 6 7 8 9\n- Reversed: 9 8 7 6 5 4 3 2 1 = 987,654,321.\n\nSo we need to compute \\(a \\times \\text{reverse}(a)\\) where \\(a = 123,456,789\\). There might be a known identity or we can compute directly using multiplication properties. \n\nWe can use the formula for multiplying a number by its reverse? Alternatively, just do standard multiplication but maybe there is a trick: \nLet \\(N = 123,456,789\\). Then \\(R(N) = 987,654,321\\). We want \\(N \\times R(N)\\). \n\nOne way: Write both numbers in terms of sums. But perhaps easier to compute using Python-like mental arithmetic? Since it's a contest problem maybe there is a neat result.\n\nLet me attempt to compute step by step:\n\nWe can use the identity:\n\\[\n(10^9 - 123,456,789) \\times (10^8 + 1,234,567,89?) \\] Not sure.\n\nAlternatively, note that \\(987,654,321 = 1000000000 - 12,345,679\\). But 12,345,679 is related to 123,456,789? Actually, 123,456,789 minus something? Let's check: \n\\(123,456,789 - 9,111,110 = ?\\) Not obvious.\n\nMaybe we can compute using the fact that:\n\\[\n987,654,321 \\times 123,456,789 = (1,000,000,000 - 12,345,679) \\times 123,456,789\n\\]\nBut then \\(12,345,679\\) is not directly related to 123,456,789.\n\nAnother idea: Use the property that for any number \\(x\\), \n\\(x \\times (10^9 - x) = 10^9 x - x^2\\). But here we have reverse, not complement.\n\nLet's just do direct multiplication using some systematic approach. Since numbers are 9-digit, product will be up to 18 digits. We can compute by splitting into parts or use known patterns: \n\\(123,456,789 \\times 987,654,321\\) is symmetric? Possibly the result might have a pattern like 121... but let's not guess.\n\nI'll attempt to compute using long multiplication with some shortcuts. Alternatively, I can break down:\n\nLet \\(A = 123,456,789\\). Write it as:\n\\(A = 123,000,000 + 456,789\\)? Not great.\n\nBetter: Use the fact that both numbers are close to powers of 10 and each other. But maybe we can compute using a calculator in mind? That's error-prone. I'll try to do step-by-step multiplication with grouping.\n\nWe want \\(A \\times B\\), where \\(B = 987,654,321\\). Notice that \\(B = 1,000,000,000 - 12,345,679\\) as earlier. So:\n\\[\nA \\times B = A \\times (10^9 - 12,345,679) = A \\times 10^9 - A \\times 12,345,679.\n\\]\nNow \\(A \\times 10^9 = 123,456,789,000,000,000\\)? Actually, \\(A = 123,456,789\\). Multiply by \\(10^9\\) gives \\(123,456,789,000,000,000\\) (since appending nine zeros). That's a huge number: 123,456,789 followed by 9 zeros.\n\nThen subtract \\(A \\times 12,345,679\\). So we need to compute \\(123,456,789 \\times 12,345,679\\). That might be simpler? Possibly not.\n\nAlternatively, use the reverse property: There is known identity for numbers like 123...n and its reverse. For n=9, maybe there's a pattern. But let's try direct multiplication with some help from known sums.\n\nI can compute using Python", "_no tool calls_"], "python": ["**Final answer:** `121932631112635269`  \n**Generated tokens:** 151  \u00b7  **Time:** 11.1 s  \u00b7  **Model calls:** 2  \u00b7  **Tool calls:** 1", "We need to compute \\(987,654,321 \\times 123,456,789\\). Using Python's arbitrary-precision integer arithmetic:\n\n```python\n987654321 * 123456789\n```\n\nThe result is `121932631112635269`.\n\nThus, the product is \\(\\boxed{121932631112635269}\\).", "**run_python #1**\n```python\n987654321 * 123456789\n```\n```text\n121932631112635269\n```"]}, {"q": "How many ways can you make change for one dollar using pennies, nickels, dimes and quarters?", "gold": "242", "none": ["**Final answer:** `(no boxed answer)`  \n**Generated tokens:** 1,024  \u00b7  **Time:** 59.4 s  \u00b7  **Model calls:** 1  \u00b7  **Tool calls:** 0  \n\u26a0\ufe0f hit the token cap", "We need to find the number of ways to make change for one dollar (100 cents) using pennies (1 cent), nickels (5 cents), dimes (10 cents), and quarters (25 cents). This is a classic coin change problem where we count the combinations of these coins that sum to 100 cents.\n\nLet:\n- \\( p \\) = number of pennies (0 or more)\n- \\( n \\) = number of nickels (0 or more)\n- \\( d \\) = number of dimes (0 or more)\n- \\( q \\) = number of quarters (0 or more)\n\nWe have the equation:\n\\[ 1p + 5n + 10d + 25q = 100 \\]\n\nWe can approach this by fixing the number of quarters (\\( q \\)), then for each, fix the number of dimes (\\( d \\)), and then count the possibilities for nickels and pennies.\n\nAlternatively, we can iterate over possible values of \\( q \\) from 0 to 4 (since \\( 25 \\times 4 = 100 \\)). For each \\( q \\), let the remaining amount be \\( R_q = 100 - 25q \\). Then we need to make \\( R_q \\) cents using nickels, dimes, and pennies.\n\nFor a fixed \\( q \\), we can iterate over possible \\( d \\) from 0 to \\( \\lfloor R_q / 10 \\rfloor \\). For each \\( d \\), let the remaining amount be \\( R_{qd} = R_q - 10d \\). Then we need to make \\( R_{qd} \\) cents using nickels and pennies. Since pennies can fill any remainder, for a given \\( n \\) (number of nickels), the number of pennies is determined as \\( p = R_{qd} - 5n \\), which must be non-negative. So for fixed \\( q \\) and \\( d \\), the number of solutions in \\( n \\) is the number of integers \\( n \\) such that:\n\\[ 0 \\leq n \\leq \\left\\lfloor \\frac{R_{qd}}{5} \\right\\rfloor \\]\nand also \\( p = R_{qd} - 5n \\geq 0 \\). Since \\( p \\) can be any non-negative integer, as long as \\( n \\leq \\lfloor R_{qd}/5 \\rfloor \\), it's valid. So the number of choices for \\( n \\) is \\( \\left\\lfloor \\frac{R_{qd}}{5} \\right\\rfloor + 1 \\).\n\nThus, for each \\( q \\) and \\( d \\), the number of ways to choose nickels and pennies is \\( \\lfloor R_{qd}/5 \\rfloor + 1 \\).\n\nNow we compute:\n\n**Case \\( q = 0 \\):**\n- \\( R_0 = 100 \\)\n- \\( d \\) from 0 to \\( \\lfloor 100/10 \\rfloor = 10 \\):\n  - For each \\( d \\), \\( R_{qd} = 100 - 10d \\).\n  - Number of nickels choices: \\( \\left\\lfloor \\frac{100 - 10d}{5} \\right\\rfloor + 1 = (20 - 2d) + 1? \\) Let's compute carefully:\n    \\[ \\frac{100 - 10d}{5} = 20 - 2d \\]\n    So floor is \\( 20 - 2d \\) (since it's integer). Then number of choices for \\( n \\) is \\( (20 - 2d) + 1 = 21 - 2d \\).\n  - But we must ensure that \\( d \\leq 10 \\), and when \\( d=10 \\), \\( R_{qd}=0 \\), then \\( \\lfloor 0/5 \\rfloor = 0 \\), so choices = 0+1=1. Our formula gives \\( 21 - 2*10 = 1 \\), correct.\n  - So for each \\( d \\) from 0 to 10, number of ways = \\( 21 - 2d \\).\n\nLet's sum over \\( d \\):\n\\[ \\sum_{d=0}^{10} (21 - 2d) \\]\nThis is an arithmetic series. Number of terms: 11.\nFirst term (\\( d=0 \\)): 21\nLast term (\\( d=10 \\)): 1\nSum = number of terms * (first + last)/2 = \\( 11 \\times (21+1)/2 = 11 \\times 22/2 = 11 \\times 11 = 121 \\).\n\n**Case \\( q = 1 \\):**\n- \\( R_1 = 100 - 25 = 75 \\)\n- \\( d \\) from 0 to \\( \\lfloor 75/10 \\rfloor = 7 \\):\n  For each \\( d \\), \\( R_{qd} = 75 - 10d \\).\n  Number of nickels choices: \\( \\left\\lfloor \\frac{75 - 10d}{5} \\right\\rfloor + 1 = (15 - 2d) + 1? \\) Compute:\n    \\[ \\frac{75 - 10d}{5", "_no tool calls_"], "python": ["**Final answer:** `242`  \n**Generated tokens:** 783  \u00b7  **Time:** 57.2 s  \u00b7  **Model calls:** 3  \u00b7  **Tool calls:** 2", "The code counts all nonnegative integers \\(q\\) (quarters), \\(d\\) (dimes), and \\(n\\) (nickels) such that the total value of those coins does not exceed 100 cents. For each such triple, the remaining amount is made up by pennies, giving exactly one way to form a dollar. The loops over quarters from 0 to 4, dimes from 0 to 10, and nickels from 0 to 20 cover all possibilities, and the condition \\(25q + 10d + 5n \\le 100\\) yields 242 valid triples.\n\nThus there are **242** different ways to make change for one dollar using pennies, nickels, dimes, and quarters.\n\n\\[\n\\boxed{242}\n\\]", "**run_python #1**\n```python\ncount = 0\nfor q in range(0, 5):  # quarters: 0 to 4 (since 5*25=125 >100)\n    for d in range(0, 11):  # dimes: 0 to floor((100-25q)/10)+1? Let's just go up to 10\n        for n in range(0, 21):  # nickels: 0 to floor((100-25q-10d)/5)+1? We'll bound by 20\n            if 25*q + 10*d + 5*n <= 100:\n                count += 1\nprint(count)\n\n```\n```text\n242\n```\n\n**run_python #2**\n```python\n# Let's verify by generating all combinations via loops and counting distinct sums? Actually we counted each triple (q,d,n) that satisfies inequality, which corresponds to a unique way including pennies. That should be correct.\n\n```\n```text\n(no output)\n```"]}, {"q": "How many primes are there below 100,000?", "gold": "9592", "none": ["**Final answer:** `9592`  \n**Generated tokens:** 217  \u00b7  **Time:** 12.7 s  \u00b7  **Model calls:** 1  \u00b7  **Tool calls:** 0", "To determine the number of prime numbers below 100,000, we can use the Prime Number Theorem or known results from prime counting functions. The exact count is given by \\(\\pi(10^5)\\), where \\(\\pi(x)\\) denotes the number of primes less than or equal to \\(x\\).\n\nFrom standard tables or computational results:\n- \\(\\pi(10^4) = 1229\\) (primes below 10,000)\n- \\(\\pi(10^5) = 9592\\)\n\nHowever, let's verify this more accurately. The Prime Number Theorem states that \\(\\pi(x) \\approx \\frac{x}{\\ln x}\\), but for exact counts up to 100,000, we can rely on known values.\n\nUpon checking reliable sources or calculating:\n- The number of primes below 10,000 is indeed 1229.\n- For 100,000, the count is 9592. This is a well-established result in number theory.\n\nThus, there are **9592** prime numbers below 100,000.\n\n\\[\n\\boxed{9592}\n\\]", "_no tool calls_"], "python": ["**Final answer:** `9592`  \n**Generated tokens:** 526  \u00b7  **Time:** 34.9 s  \u00b7  **Model calls:** 2  \u00b7  **Tool calls:** 1", "The Python code computed the number of primes below 100,000 using the Sieve of Eratosthenes. The result is **9592**.\n\nThus, there are \\(\\boxed{9592}\\) primes below 100,000.", "**run_python #1**\n```python\ndef count_primes(limit):\n    sieve = [True] * (limit+1)\n    sieve[0] = sieve[1] = False\n    for i in range(2, int(limit**0.5) + 1):\n        if sieve[i]:\n            step = i\n            start = i*i\n            # mark multiples of i starting from i*i as False\n            for j in range(start, limit+1, step):\n                sieve[j] = False\n    return sum(sieve)\n\nprint(count_primes(100000))\n\n```\n```text\n9592\n```"]}]; const REC="### Recorded results\n\nThe main win is fewer generated tokens and less time at about the same accuracy. Accuracy rose on AIME 2025 but not significantly for the base prompt. On MATH-500 level 5 it went down slightly.\n\n| Set | Model | No tools | run_python | Tokens / problem | Change |\n|---|---|---|---|---|---|\n| MATH (30) | LFM3-1.7B | 30/30 | 29/30 | 3,036.8 \u2192 2,022.8 | \u221233.4% |\n| MATH (30) | Clover-35B-A3B | 28/30 | 28/30 | 2,791.7 \u2192 1,458.6 | \u221247.8% |\n| AIME 2025 (30) | LFM3-1.7B | 17/30 | 20/30 | 16,634.1 \u2192 9,595.6 | \u221242.3% |\n| MATH-500 L5 (60) | LFM3-1.7B | 54/60 | 50/60 | 7,577.1 \u2192 5,971.6 | \u221221.2% |\n| Combined (90) | LFM3-1.7B | 71/90 | 70/90 | 10,596.1 \u2192 7,179.6 | \u221232.2% |\n\nTime per problem on MATH (30): LFM3 24.18 s \u2192 16.64 s (\u221231.2%). Clover 131.68 s \u2192 69.44 s (\u221247.3%). I only quote tokens for AIME and L5, because part of the no-tools side ran with parallel workers on a shared server.\n\nAccuracy (Wilson 95% CI, McNemar exact p):\n\n- LFM3 on MATH: 30/30 (88.6\u2013100%) without tools. 29/30 with Python by the grader, 30/30 by hand. One answer was a formatting miss.\n- Clover on MATH: 28/30 both ways, p = 1.0.\n- AIME 2025: 17/30 (39.2\u201372.6%) \u2192 20/30 (48.8\u201380.8%), +10.0 pts. Fixed 4, broke 1, p = 0.375. Not significant. Token-cap hits 11 \u2192 2.\n- MATH-500 L5: 54/60 (79.9\u201395.3%) \u2192 50/60 (72.0\u201390.7%), \u22126.7 pts. Fixed 0, broke 4, p = 0.125.\n- Combined 90: fixed 4, broke 5, p = 1.0. Token-cap hits 14 \u2192 3.\n- `--verify` variant on AIME: 17/30 \u2192 24/30 (62.7\u201390.5%), +23.3 pts, fixed 8, broke 1, p = 0.039. Tokens \u221229.8%. This is preliminary and post-hoc. I tried it after seeing the base results, and its L5 run had only 3 of 60 done at publish time.\n\nSetup: llama.cpp `llama-server`, GGUF, greedy, thinking on, 25,000 generated-token budget per problem summed across all model calls. Up to 6 `run_python` calls, 10 s timeout per call, output capped at 2,000 characters. Apple M1 Max.\n\nRaw data, the benchmark script and the grader: https://huggingface.co/austindixson/python-sandbox-math-tool\n\n### Demo runs on 2 CPU threads (same app, 1,024-token cap)\n\nRecorded before publishing, on a 2-thread Linux machine (bwrap sandbox) with this Space's own `app.py`. These are demo-setting numbers, not the eval above.\n\n| Question | Correct answer | No tools | run_python |\n|---|---|---|---|\n| What is the sum of the digits of 2^1000? | 1366 | (no boxed answer) \u00b7 1,024 tok \u00b7 61.5 s | 1366 \u00b7 153 tok \u00b7 12.3 s |\n| What is the remainder when 7^2026 is divided by 1000? | 649 | (no boxed answer) \u00b7 1,024 tok \u00b7 61.4 s | 649 \u00b7 281 tok \u00b7 19.8 s |\n| What is 987654321 \u00d7 123456789? | 121932631112635269 | (no boxed answer) \u00b7 1,024 tok \u00b7 61.7 s | 121932631112635269 \u00b7 151 tok \u00b7 11.1 s |\n| How many ways can you make change for one dollar using pennies, nickels, dimes and quarters? | 242 | (no boxed answer) \u00b7 1,024 tok \u00b7 59.4 s | 242 \u00b7 783 tok \u00b7 57.2 s |\n| How many primes are there below 100,000? | 9592 | 9592 \u00b7 217 tok \u00b7 12.7 s | 9592 \u00b7 526 tok \u00b7 34.9 s |\n\nA sixth question I tried (integers up to 10,000 divisible by 3 or 7 but not 5) hit the 1,024-token cap both ways without calling the tool, so I left it out of the presets.\n";
+const md=s=>marked.parse(s);
+function tab(n){for(const i of [1,2]){document.getElementById('p'+i).style.display=i==n?'':'none';document.getElementById('t'+i).className=i==n?'on':''}}
+const sel=document.getElementById('q');
+RUNS.forEach((r,i)=>{const o=document.createElement('option');o.value=i;o.textContent=r.q+'  (correct: '+r.gold+')';sel.appendChild(o)});
+function show(){const r=RUNS[sel.value];
+ document.getElementById('ns').innerHTML=md(r.none[0]);document.getElementById('nt').textContent=r.none[1]||'(empty)';
+ document.getElementById('ps').innerHTML=md(r.python[0]);document.getElementById('pt').textContent=r.python[1]||'(empty)';
+ document.getElementById('pc').innerHTML=md(r.python[2]);}
+document.getElementById('rec').innerHTML=md(REC); show();
+</script></body></html>

live/app.py

@@ -0,0 +1,125 @@
+"""Side-by-side demo: the same small model answers a math question with no tools and with a
+sandboxed run_python tool. Runs llama.cpp's llama-server on the Space's free CPU."""
+import os, re, json, time, tarfile, subprocess, urllib.request, threading, glob
+import gradio as gr
+from huggingface_hub import hf_hub_download
+import sandbox
+from proxy import chat_with_python
+
+LLAMA_TAG = "b11938"
+MODEL_REPO, MODEL_FILE = "LiquidAI/LFM3-1.7B-GGUF", "LFM3-1.7B-QAD-Q4_0.gguf"
+PORT = 8080
+URL = os.environ.get("LLAMA_URL", f"http://127.0.0.1:{PORT}")
+TOOL_REPO = "https://huggingface.co/austindixson/python-sandbox-math-tool"
+SYS = "Solve the math problem. Reason step by step, then put the final answer in \\boxed{}."
+_ready = threading.Event(); _status = {"msg": "starting"}
+
+def _start_server():
+    if os.environ.get("LLAMA_URL"):
+        _ready.set(); return
+    try:
+        d = "/tmp/llama"
+        if not glob.glob(f"{d}/**/llama-server", recursive=True):
+            _status["msg"] = "downloading llama.cpp"
+            os.makedirs(d, exist_ok=True)
+            tgz = f"{d}/llama.tgz"
+            urllib.request.urlretrieve(f"https://github.com/ggml-org/llama.cpp/releases/download/{LLAMA_TAG}/llama-{LLAMA_TAG}-bin-ubuntu-x64.tar.gz", tgz)
+            with tarfile.open(tgz) as t: t.extractall(d)
+        srv = glob.glob(f"{d}/**/llama-server", recursive=True)[0]
+        _status["msg"] = "downloading model"
+        gguf = hf_hub_download(MODEL_REPO, MODEL_FILE)
+        _status["msg"] = "loading model"
+        env = dict(os.environ, LD_LIBRARY_PATH=os.path.dirname(srv))
+        subprocess.Popen([srv, "-m", gguf, "-c", "8192", "-np", "1", "-t", os.environ.get("LLAMA_THREADS", "2"),
+                          "--host", "127.0.0.1", "--port", str(PORT), "--jinja"], env=env,
+                         stdout=open("/tmp/llama.log", "w"), stderr=subprocess.STDOUT, cwd=os.path.dirname(srv))
+        for _ in range(300):
+            try:
+                if json.load(urllib.request.urlopen(URL + "/health", timeout=2)).get("status") == "ok":
+                    _ready.set(); _status["msg"] = "ready"; return
+            except Exception: pass
+            time.sleep(1)
+        _status["msg"] = "model server did not start: " + open("/tmp/llama.log").read()[-500:]
+    except Exception as e:
+        _status["msg"] = f"startup failed: {e}"
+threading.Thread(target=_start_server, daemon=True).start()
+
+EXAMPLES = [
+    "What is the sum of the digits of 2^1000?",
+    "What is the remainder when 7^2026 is divided by 1000?",
+    "What is 987654321 × 123456789?",
+    "How many ways can you make change for one dollar using pennies, nickels, dimes and quarters?",
+    "How many primes are there below 100,000?",
+]
+
+def _boxed(s):
+    i = s.rfind("\\boxed{")
+    if i < 0: return None
+    j, d = i + 7, 1
+    for k in range(j, len(s)):
+        d += {"{": 1, "}": -1}.get(s[k], 0)
+        if d == 0: return s[j:k]
+    return None
+
+def _one(q, use_tool, max_tokens):
+    body = {"model": "lfm3", "max_tokens": int(max_tokens), "temperature": 0.0, "top_k": 1,
+            "chat_template_kwargs": {"enable_thinking": False},
+            "messages": [{"role": "system", "content": SYS}, {"role": "user", "content": q}]}
+    t = time.time(); r = chat_with_python(URL, body, use_tool=use_tool); dt = time.time() - t
+    m = r["choices"][0]["message"]; txt = re.sub(r"<think>.*?</think>", "", m.get("content") or "", flags=re.S).strip()
+    x = r["x_python_tool"]; fin = r["choices"][0].get("finish_reason")
+    stats = (f"**Final answer:** `{_boxed(txt) or '(no boxed answer)'}`  \n"
+             f"**Generated tokens:** {r['usage']['completion_tokens']:,}  ·  **Time:** {dt:.1f} s  ·  "
+             f"**Model calls:** {x['llm_calls']}  ·  **Tool calls:** {len(x['calls'])}"
+             + ("  \n⚠️ hit the token cap" if fin == "length" else ""))
+    calls = "\n\n".join(f"**run_python #{i}**\n```python\n{c['code']}\n```\n```text\n{c['output']}\n```" for i, c in enumerate(x["calls"], 1)) or "_no tool calls_"
+    return stats, txt, calls
+
+def compare(q, max_tokens):
+    if not q.strip():
+        yield "Type a question first.", "", "", "", "", ""; return
+    if not _ready.wait(timeout=1):
+        yield f"Model is still loading ({_status['msg']}). Try again in a minute.", "", "", "", "", ""; return
+    wait = "_running..._"
+    yield wait, "", "", "_waiting for the no-tools run to finish..._", "", ""
+    a = _one(q, False, max_tokens)
+    yield a[0], a[1], "", wait, "", ""
+    b = _one(q, True, max_tokens)
+    yield a[0], a[1], "", b[0], b[1], b[2]
+
+RECORDED = open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "RECORDED.md")).read()
+
+backend = sandbox.pick_backend()
+with gr.Blocks(title="Python sandbox tool: with vs without") as demo:
+    gr.Markdown(f"""# Small model, with and without a Python tool
+The same model (LiquidAI **LFM3-1.7B**, QAD-Q4_0 GGUF, llama.cpp {LLAMA_TAG}, greedy, thinking off) answers your
+question twice: once on its own, once with a `run_python` tool it can call up to 6 times. Runs on this Space's
+free 2-vCPU CPU, so expect roughly 10–60 s per side. Tool code: [{TOOL_REPO.split('co/')[1]}]({TOOL_REPO}).
+
+**Sandbox here:** `{backend}`: {sandbox.ISOLATION.get(backend)}. On a Mac the tool uses `sandbox-exec`
+(network denied by the OS, writes only in a temp dir); on Linux with working user namespaces it uses bubblewrap.""")
+    with gr.Tab("Try it"):
+        with gr.Row():
+            q = gr.Textbox(label="Math question", value=EXAMPLES[0], lines=2, scale=4)
+            mt = gr.Slider(256, 2048, value=1024, step=128, label="Token cap per answer (demo only)", scale=1)
+        gr.Examples(EXAMPLES, inputs=q)
+        go = gr.Button("Run both", variant="primary")
+        with gr.Row():
+            with gr.Column():
+                gr.Markdown("### No tools")
+                s1 = gr.Markdown(); a1 = gr.Markdown(); c1 = gr.Markdown(visible=False)
+            with gr.Column():
+                gr.Markdown("### With run_python")
+                s2 = gr.Markdown(); a2 = gr.Markdown()
+                with gr.Accordion("Tool calls and outputs", open=True):
+                    c2 = gr.Markdown()
+        go.click(compare, [q, mt], [s1, a1, c1, s2, a2, c2], concurrency_limit=1)
+        gr.Markdown("The token cap here is a demo setting. Every recorded benchmark run used a 25,000-token budget per problem.")
+    with gr.Tab("Recorded results"):
+        gr.Markdown(RECORDED)
+        with gr.Row():
+            gr.Image("fig_tokens.png", label="Tokens per problem", show_label=True)
+            gr.Image("fig_aime.png", label="Hard math (AIME 2025)", show_label=True)
+
+if __name__ == "__main__":
+    demo.queue(default_concurrency_limit=1).launch()

live/packages.txt

@@ -0,0 +1,2 @@
+libgomp1
+bubblewrap

live/proxy.py

@@ -0,0 +1,138 @@
+"""proxy.py - OpenAI-compatible proxy that gives any local model a sandboxed run_python tool.
+
+    python proxy.py --upstream http://127.0.0.1:8080 --port 8099
+    # then point your client at http://127.0.0.1:8099/v1
+
+For each /v1/chat/completions request it adds the run_python tool, runs every run_python
+call in the sandbox (see sandbox.py), feeds the output back, and repeats until the model
+answers or it hits --max-tool-calls (then it tells the model to answer now). You get back
+one normal chat completion. usage is summed over all upstream calls, and an extra
+"x_python_tool" field lists each call with its code and output.
+
+If the client sends max_tokens, it is treated as the total budget for the whole loop
+(each upstream call gets what is left), so runs with and without the tool are comparable.
+Only the Python standard library is used.
+"""
+import argparse, json, time, urllib.request
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+import sandbox
+
+SYSTEM_HINT = (" You have a run_python tool (max {n} calls) that executes Python 3 code and returns "
+               "stdout; use it to compute or verify.")
+VERIFY_HINT = (" Before you give the final answer, you must run Python code that independently checks it "
+               "(substitute back, brute-force small cases, or recompute another way). Only box an answer that your code confirmed.")
+NUDGE = ("Tool budget used up ({n}/{n}). Do not call any more tools. Give your final answer now.")
+
+def _post(upstream, body, timeout=7200):
+    req = urllib.request.Request(upstream.rstrip("/") + "/v1/chat/completions", data=json.dumps(body).encode(),
+                                 headers={"Content-Type": "application/json"})
+    return json.load(urllib.request.urlopen(req, timeout=timeout))
+
+def chat_with_python(upstream, body, max_tool_calls=6, add_hint=True, use_tool=True, on_event=None, verify=False):
+    """Run one chat request with the run_python loop. Returns an OpenAI chat.completion dict.
+    use_tool=False sends the same request with no tool (handy for A/B runs).
+    verify=True also asks the model to check its answer with code before boxing it.
+    on_event(kind, data) is called for 'llm' and 'tool' steps if given."""
+    body = dict(body); body.pop("stream", None)
+    msgs = [dict(m) for m in body.get("messages", [])]
+    client_tools = [t for t in body.get("tools") or [] if t.get("function", {}).get("name") != "run_python"]
+    if use_tool and add_hint:
+        hint = SYSTEM_HINT.format(n=max_tool_calls) + (VERIFY_HINT if verify else "")
+        if msgs and msgs[0].get("role") == "system":
+            msgs[0]["content"] = (msgs[0].get("content") or "") + hint
+        else:
+            msgs.insert(0, {"role": "system", "content": hint.strip()})
+    budget = body.get("max_tokens") or body.get("max_completion_tokens")
+    used = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
+    calls, steps, nudged, peak_ctx = [], [], False, 0
+    t0 = time.time(); r = None
+    for _ in range(max_tool_calls + 2):
+        req = dict(body); req["messages"] = msgs
+        if budget:
+            left = budget - used["completion_tokens"]
+            if left <= 0: break
+            req["max_tokens"] = left; req.pop("max_completion_tokens", None)
+        tools = list(client_tools)
+        if use_tool and len(calls) < max_tool_calls:
+            tools.append(sandbox.TOOL_SCHEMA)
+        elif use_tool and not nudged:
+            msgs.append({"role": "user", "content": NUDGE.format(n=max_tool_calls)}); nudged = True
+        if tools: req["tools"] = tools
+        else: req.pop("tools", None); req.pop("tool_choice", None)
+        r = _post(upstream, req)
+        u = r.get("usage") or {}
+        for k in used: used[k] += u.get(k) or 0
+        peak_ctx = max(peak_ctx, (u.get("prompt_tokens") or 0) + (u.get("completion_tokens") or 0))
+        ch = r["choices"][0]; m = ch["message"]
+        steps.append({"completion_tokens": u.get("completion_tokens"), "finish_reason": ch.get("finish_reason")})
+        if on_event: on_event("llm", {"message": m, "usage": u})
+        tc = m.get("tool_calls") or []
+        mine = [c for c in tc if c.get("function", {}).get("name") == "run_python"]
+        if not mine or len(mine) != len(tc) or ch.get("finish_reason") == "length":
+            break  # final answer, or a client tool the caller must handle
+        am = {"role": "assistant", "content": m.get("content") or "", "tool_calls": tc}
+        if m.get("reasoning_content"): am["reasoning_content"] = m["reasoning_content"]
+        msgs.append(am)
+        for c in mine:
+            try: code = json.loads(c["function"].get("arguments") or "{}").get("code", "")
+            except Exception: code = ""
+            out = sandbox.run_python(code)
+            calls.append({"code": code, "output": out})
+            if on_event: on_event("tool", calls[-1])
+            msgs.append({"role": "tool", "tool_call_id": c.get("id", ""), "content": out})
+    if r is None:
+        r = {"object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": ""}, "finish_reason": "length"}]}
+    r["usage"] = used
+    r["x_python_tool"] = {"backend": sandbox.pick_backend(), "calls": calls, "llm_calls": len(steps), "steps": steps,
+                          "peak_context_tokens": peak_ctx, "seconds": round(time.time() - t0, 2)}
+    return r
+
+class Handler(BaseHTTPRequestHandler):
+    upstream = None; max_tool_calls = 6; add_hint = True; verify = False
+    def _send(self, code, data, ctype="application/json"):
+        self.send_response(code); self.send_header("Content-Type", ctype)
+        self.send_header("Content-Length", str(len(data))); self.end_headers(); self.wfile.write(data)
+    def _forward(self, method, raw=None):
+        req = urllib.request.Request(self.upstream.rstrip("/") + self.path, data=raw, method=method,
+                                     headers={"Content-Type": "application/json"})
+        try:
+            with urllib.request.urlopen(req, timeout=600) as f: self._send(f.status, f.read())
+        except urllib.error.HTTPError as e: self._send(e.code, e.read())
+    def do_GET(self): self._forward("GET")
+    def do_POST(self):
+        raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
+        if not self.path.rstrip("/").endswith("/chat/completions"):
+            return self._forward("POST", raw)
+        try:
+            body = json.loads(raw)
+            r = chat_with_python(self.upstream, body, self.max_tool_calls, self.add_hint, verify=self.verify)
+        except urllib.error.HTTPError as e:
+            return self._send(e.code, e.read())
+        except Exception as e:
+            return self._send(500, json.dumps({"error": {"message": str(e)}}).encode())
+        if body.get("stream"):  # send the finished answer as a single SSE chunk
+            m = r["choices"][0]["message"]
+            chunk = {"id": r.get("id", "x"), "object": "chat.completion.chunk", "created": r.get("created", int(time.time())),
+                     "model": r.get("model", body.get("model", "")),
+                     "choices": [{"index": 0, "delta": {k: v for k, v in m.items() if k in ("role", "content", "reasoning_content", "tool_calls")},
+                                  "finish_reason": r["choices"][0].get("finish_reason")}], "usage": r["usage"]}
+            data = f"data: {json.dumps(chunk)}\n\ndata: [DONE]\n\n".encode()
+            return self._send(200, data, "text/event-stream")
+        self._send(200, json.dumps(r).encode())
+    def log_message(self, fmt, *a): pass
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    ap.add_argument("--upstream", required=True, help="base URL of your model server, e.g. http://127.0.0.1:8080")
+    ap.add_argument("--host", default="127.0.0.1"); ap.add_argument("--port", type=int, default=8099)
+    ap.add_argument("--max-tool-calls", type=int, default=6)
+    ap.add_argument("--no-hint", action="store_true", help="don't append the tool hint to the system prompt")
+    ap.add_argument("--verify", action="store_true", help="also tell the model to check its answer with code (best AIME result so far)")
+    a = ap.parse_args()
+    Handler.upstream, Handler.max_tool_calls, Handler.add_hint, Handler.verify = a.upstream, a.max_tool_calls, not a.no_hint, a.verify
+    b = sandbox.pick_backend()
+    print(f"python-tool proxy on http://{a.host}:{a.port}/v1 -> {a.upstream}  sandbox={b} ({sandbox.ISOLATION[b]})", flush=True)
+    ThreadingHTTPServer((a.host, a.port), Handler).serve_forever()
+
+if __name__ == "__main__":
+    main()

live/requirements.txt

@@ -0,0 +1,4 @@
+huggingface_hub
+sympy
+numpy
+gradio==6.31.0

live/sandbox.py

@@ -0,0 +1,146 @@
+"""sandbox.py - run untrusted Python for a model's run_python tool.
+
+One function you need:  run_python(code) -> str
+
+Backends (picked automatically, or set PYSANDBOX_BACKEND):
+  macos     sandbox-exec profile: no network, file writes only inside a fresh temp dir.  (strong)
+  bwrap     Linux bubblewrap: new net/pid/ipc namespaces (no network), read-only root,
+            only the temp dir is writable, empty env.                               (strong)
+  fallback  plain subprocess: rlimits (CPU, memory, file size), empty env, temp cwd,
+            socket module blocked inside the child. Python-level only, a determined
+            script can get around it. Linux/macOS without the above, and Windows.    (weak)
+
+Every backend: 10 s wall timeout, isolated interpreter (-I -B), last bare expression is
+printed (like a REPL), output capped at 2000 chars.
+"""
+import os, sys, shutil, subprocess, tempfile, platform
+
+TIMEOUT = 10
+MAX_OUT = 2000
+
+TOOL_SCHEMA = {"type": "function", "function": {
+    "name": "run_python",
+    "description": "Execute Python 3 code in a sandbox (no network, 10 s timeout). Returns stdout; the value of a final bare expression is printed.",
+    "parameters": {"type": "object", "properties": {"code": {"type": "string", "description": "Python code"}}, "required": ["code"]}}}
+
+# Reads the code from stdin, prints the last bare expression, runs it.
+_WRAPPER = (
+    "import ast,sys\n"
+    "src=sys.stdin.read()\n"
+    "t=ast.parse(src)\n"
+    "g={'__name__':'__main__'}\n"
+    "last=t.body.pop() if t.body and isinstance(t.body[-1],ast.Expr) else None\n"
+    "exec(compile(t,'<tool>','exec'),g)\n"
+    "if last is not None:\n"
+    "    v=eval(compile(ast.Expression(last.value),'<tool>','eval'),g)\n"
+    "    if v is not None: print(v)\n")
+
+# Only used by the weak fallback: make every socket call fail inside the child.
+_SOCKET_BLOCK = (
+    "import socket as _s\n"
+    "def _deny(*a,**k): raise PermissionError('network disabled in sandbox')\n"
+    "class _NoSock:\n"
+    "    def __init__(self,*a,**k): _deny()\n"
+    "_s.socket=_NoSock; _s.create_connection=_deny; _s.getaddrinfo=_deny; _s.socketpair=_deny; _s.fromfd=_deny\n"
+    "import _socket as _r; _r.socket=_NoSock; _r.getaddrinfo=_deny; _r.socketpair=_deny\n"
+    "del _s,_r,_NoSock\n"
+    "import builtins as _b, os as _o, sys as _y\n"
+    "_BLOCK={'ctypes','_ctypes','subprocess','multiprocessing','_posixsubprocess','ssl','_ssl','asyncio','selectors','select','requests','httpx','pty','resource'}\n"
+    "_imp=_b.__import__\n"
+    "def _guard(name,*a,**k):\n"
+    "    if name.split('.')[0] in _BLOCK and _y._getframe(1).f_code.co_filename=='<tool>': raise ImportError(f'module {name!r} is disabled in this sandbox')\n"
+    "    return _imp(name,*a,**k)\n"
+    "_b.__import__=_guard\n"
+    "for _n in ('system','popen','fork','forkpty','kill','killpg','execv','execve','execl','execlp','execvp','execvpe','spawnv','spawnve','spawnl','posix_spawn','posix_spawnp'):\n"
+    "    if hasattr(_o,_n): delattr(_o,_n)\n"
+    "del _b,_o,_n\n")
+# Note: the import guard only blocks imports written in the tool code itself, so libraries such
+# as numpy that use ctypes internally still load. It is a speed bump, not a security boundary.
+
+def _macos_profile(workdir):
+    w = os.path.realpath(workdir)
+    return ("(version 1)(allow default)(deny network*)"
+            "(deny file-write*)"
+            f'(allow file-write* (subpath "{w}") (literal "/dev/null") (literal "/dev/zero")'
+            ' (literal "/dev/stdout") (literal "/dev/stderr") (literal "/dev/tty") (regex #"^/dev/fd/"))')
+
+_bwrap_ok = None
+def _bwrap_works():
+    """bwrap is often installed but unusable (no user namespaces in many containers)."""
+    global _bwrap_ok
+    if _bwrap_ok is None:
+        b = shutil.which("bwrap")
+        _bwrap_ok = False
+        if b:
+            try:
+                r = subprocess.run([b, "--unshare-all", "--ro-bind", "/", "/", "--dev", "/dev", "true"],
+                                   capture_output=True, timeout=5)
+                _bwrap_ok = r.returncode == 0
+            except Exception:
+                pass
+    return _bwrap_ok
+
+def pick_backend():
+    forced = os.environ.get("PYSANDBOX_BACKEND")
+    if forced:
+        return forced
+    s = platform.system()
+    if s == "Darwin" and shutil.which("sandbox-exec"):
+        return "macos"
+    if s == "Linux" and _bwrap_works():
+        return "bwrap"
+    return "fallback"
+
+ISOLATION = {"macos": "strong (sandbox-exec: network denied, writes only in temp dir)",
+             "bwrap": "strong (bubblewrap: no network namespace, read-only root, temp dir writable)",
+             "fallback": "weak (rlimits + empty env + temp cwd + Python-level socket block; use Docker/WSL for real isolation)"}
+
+def _limits():
+    import resource
+    def f():
+        resource.setrlimit(resource.RLIMIT_CPU, (TIMEOUT, TIMEOUT))
+        for name, val in (("RLIMIT_AS", 4 << 30), ("RLIMIT_FSIZE", 64 << 20)):
+            try: resource.setrlimit(getattr(resource, name), (val, val))
+            except (ValueError, OSError, AttributeError): pass
+        os.setsid()
+    return f
+
+def run_python(code, python=None, backend=None, timeout=TIMEOUT):
+    """Run `code`, return stdout (plus stderr if it failed), capped at MAX_OUT chars."""
+    python = python or os.environ.get("PYSANDBOX_PYTHON") or sys.executable
+    backend = backend or pick_backend()
+    with tempfile.TemporaryDirectory(prefix="pysbx-") as d:
+        env = {"PATH": "/usr/bin:/bin", "HOME": d, "TMPDIR": d, "PYTHONDONTWRITEBYTECODE": "1",
+               "OMP_NUM_THREADS": "1", "OPENBLAS_NUM_THREADS": "1", "MKL_NUM_THREADS": "1"}
+        kw = dict(input=code, capture_output=True, text=True, errors="replace", timeout=timeout, cwd=d, env=env)
+        if backend == "macos":
+            cmd = ["sandbox-exec", "-p", _macos_profile(d), python, "-I", "-B", "-c", _WRAPPER]
+        elif backend == "bwrap":
+            py = python  # keep the venv path (not realpath) so the venv's site-packages are used
+            cmd = ["bwrap", "--unshare-all", "--die-with-parent", "--new-session",
+                   "--ro-bind", "/", "/", "--dev", "/dev", "--proc", "/proc", "--tmpfs", "/tmp",
+                   "--bind", d, d, "--chdir", d, "--clearenv",
+                   "--setenv", "PATH", "/usr/bin:/bin", "--setenv", "HOME", d, "--setenv", "OPENBLAS_NUM_THREADS", "1",
+                   py, "-I", "-B", "-c", _WRAPPER]
+        elif backend == "fallback":
+            cmd = [python, "-I", "-B", "-c", _SOCKET_BLOCK + _WRAPPER]
+            if os.name == "nt":
+                env.update({"SYSTEMROOT": os.environ.get("SYSTEMROOT", r"C:\Windows")})
+                env["PATH"] = os.path.dirname(python)
+                kw["creationflags"] = 0x08000000  # CREATE_NO_WINDOW
+            else:
+                kw["preexec_fn"] = _limits()
+        else:
+            raise ValueError(f"unknown backend {backend!r}")
+        try:
+            p = subprocess.run(cmd, **kw)
+            out = (p.stdout + ("\nSTDERR: " + p.stderr[-800:] if p.returncode else "")).strip()
+        except subprocess.TimeoutExpired:
+            out = f"ERROR: timeout after {timeout} s"
+    return (out or "(no output)")[:MAX_OUT]
+
+if __name__ == "__main__":
+    b = pick_backend()
+    print(f"backend: {b} - {ISOLATION.get(b, '?')}")
+    src = sys.stdin.read() if not sys.stdin.isatty() else "import math\nmath.factorial(30)"
+    print(run_python(src, backend=b))