Spaces:
Running
Running
File size: 31,489 Bytes
3c79d0e 739eecc f093ab7 baf8832 b550fec 2a8c072 3c79d0e b550fec 5baf7d1 b550fec 3c79d0e 16483b0 3c79d0e 16483b0 3c79d0e 16483b0 3c79d0e 1a60da1 9167d2e 2362166 a5990d2 3c79d0e 16483b0 3c79d0e 16483b0 3c79d0e 16483b0 3c79d0e 2362166 3c79d0e 1a60da1 16483b0 3c79d0e 16483b0 9167d2e 16483b0 3c79d0e 16483b0 3c79d0e 16483b0 3c79d0e a5990d2 3c79d0e 1bf1db3 2362166 9167d2e 3c79d0e 1a60da1 2362166 9167d2e a5990d2 9167d2e 3c79d0e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 | <!doctype html><html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Floorplan Graph RL: Beam reads real floor plans by writing code</title><style>body{font-family:system-ui,-apple-system,sans-serif;max-width:1100px;margin:2em auto;padding:0 1em;line-height:1.5;color:#222}table{border-collapse:collapse;margin:1em 0}td,th{border:1px solid #ccc;padding:6px 10px;text-align:right}td:first-child,th:first-child{text-align:left}.figs{display:grid;grid-template-columns:1fr 1fr;gap:12px}.figs img{width:100%;border:1px solid #ddd}.cap{font-size:.9em;color:#444;margin-bottom:1.5em}code{background:#f3f3f3;padding:1px 4px}.note{color:#666;font-size:.9em}details{margin:.6em 0;border:1px solid #e3e3e3;border-radius:6px;padding:.4em .8em}summary{cursor:pointer;font-size:1.1em}</style></head><body>
<h1>Floorplan Graph RL: Beam reads real apartment floor plans by writing code</h1>
<p style='font-size:1.3em'><b>Beam can't see an image. So it learned to write its own eyes.</b></p>
<p style='font-size:1.15em'><b>A text-only Beam agent writes its own computer-vision pipeline, learns from exact grader feedback, and improves without weight updates.</b> Its best 10-round run reaches <b>0.179 held-out partial credit</b> on 64 unseen real apartment plans and <b>0.803 room F1 on clean plans</b>.</p><p><b>Beam actually learned:</b> after the grader reported zero rooms on scans, Beam diagnosed its thresholding failure and adopted adaptive thresholding. Public partial credit jumped from <b>0.022 to 0.179</b>.</p><p><b>The problem.</b> Many floor plans exist only as images or scans. A room graph (rooms + doors) answers "can a wheelchair reach every room?", "how do I get out?" and "how many rooms?". This environment teaches a text-only model to build that graph from a real plan, and grades it exactly.</p><p><b>In one screen.</b></p>
<ul>
<li><b>A novel Beam loop:</b> Beam writes a complete vision program, the environment executes it on real plans, and Beam uses exact rewards and failure reports to revise it. Every prompt, reasoning trace, program, and score is published.</li>
<li><b>Measured improvement:</b> the 10-round run rises from <b>0.000 to 0.182 public partial credit</b>; the selected round scores <b>0.179 on 64 held-out plans from other buildings</b>. A 46-line general tips playbook also lifts Beam low from <b>0.032 to 0.075</b> held-out (3 repeats each; wide spread).</li>
<li><b>A reusable, unsaturated RL environment:</b> <code>reset()</code> / <code>step(code)</code>, strict and dense rewards, 24 practice + 64 held-out plans, Docker grading, and a GRPO example. Strict reward remains <b>0.000</b> versus <b>0.242</b> for the reference: rooms are emerging; doors are the open research problem.</li>
</ul><p>Data, grader, Docker files and every run: <a href='https://huggingface.co/datasets/hamoodey/floorplan-graph-rl'>huggingface.co/datasets/hamoodey/floorplan-graph-rl</a></p><p><img src="img/hero.png" style="width:100%" alt="input plan, true room graph, Beam graph, route to exit"></p><p class="note">Plan p17 (clean style). Panel 3: rooms recovered by Beam's own program (Beam low, round 9). Panel 4: the step-free route to the exit, a networkx shortest path over the true door graph; this is what a recovered graph is for. Beam's doors are not right yet (door F1 0), so its own route would not be trustworthy.</p><table><tr><th>Evidence</th><th>Result</th></tr><tr><td>Beam low, 10 feedback rounds</td><td><b>0.000 → 0.182 public partial</b>; <b>0.179 held-out partial</b></td></tr><tr><td>General tips playbook, 3 repeats each</td><td><b>0.032 → 0.075 held-out partial</b></td></tr><tr><td>Clean-plan room recovery, best 10-round run</td><td><b>0.803 room F1</b></td></tr><tr><td>Strict held-out reward</td><td><b>0.000</b> for every Beam setup; <b>0.242</b> reference</td></tr></table><div style="display:flex;gap:16px;flex-wrap:wrap;align-items:flex-start"><img src="img/rounds.gif" style="max-width:460px;width:100%;border:1px solid #ddd" alt="Beam rooms per round"><div style="flex:1;min-width:300px"><img src="img/bars.png" style="width:100%" alt="held-out partial credit by setup"></div></div><p><img src="img/rounds-chart.png" style="width:100%" alt="score per round"></p><h2>Use it for RL</h2><pre style="background:#f6f6f6;padding:1em;overflow-x:auto"><code>from code.env import FloorplanEnv, extract_code
env = FloorplanEnv(set="public") # 24 practice plans; "hidden" only for the final score
prompt = env.reset() # instruction + contract + style guide
reward, partial, feedback, info = env.step(extract_code(model_reply))
next_prompt = prompt + previous_code + feedback # the in-context loop used here
def floorplan_reward(completions, **kw): # GRPO-style reward (e.g. TRL GRPOTrainer)
return [sum(FloorplanEnv("public").step(extract_code(c if isinstance(c, str) else c[-1]["content"]))[:2])
for c in completions] # strict + dense partial credit</code></pre><p>Unsaturated: strict reward is still 0.000 for every model loop here, while the hand-written reference reaches 0.242. Lots of room to train.</p>
<h2>What generalizes for anyone using Beam on code-writing tasks</h2>
<ul>
<li>Feedback beats a single try: graded retries took Beam from 0.000 (round 1) to 0.179 held-out partial credit with no weight change.</li>
<li>Write down what worked: a 46-line tips playbook more than doubled the held-out score (0.032 → 0.075, 3 repeats each).</li>
<li>Grade on data the feedback never saw: 64 held-out plans from other buildings keep the gain honest.</li>
<li>Score partial credit as well as strict: strict was 0.000 in every run, so only partial credit shows learning.</li>
</ul><h2>Beam reading its failure report (<code>reasoning_content</code>, Beam low, 10-round run)</h2><blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em"><b>Round 5</b> (practice-plan partial credit 0.022 → 0.179): “My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach.”</blockquote><blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em"><b>Round 7</b> (practice-plan partial credit 0.078 → 0.174): “The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet.”</blockquote><h2>Why trust these numbers</h2>
<p>The grader is exact (rooms at IoU 0.7, doors within 0.5 m). Feedback uses only the 24 practice plans; the 64 held-out plans are from other buildings and never feed back. Every prompt, reasoning trace, program and score is in <code>runs/</code>.</p>
<p><b>Known limits.</b> Kimi K3 did not finish (no reply to its round-1 request in 31 min); Beam high effort finished (held-out 0.016 in a single 3-round run; 0.176 with best of 4 samples per round). Most numbers are one run; the playbook result has 3 repeats with a wide spread. Doors are nearly unsolved (door F1 0.000 on every setup but one). We did not update Beam's weights.</p><details><summary><b>What this entry is</b></summary><ul><li><b>An RL environment for Beam.</b> <code>FloorplanEnv.reset()</code> gives the task prompt; <code>step(code)</code> runs Beam's <code>run.py</code> and returns the strict reward, a dense partial-credit reward and per-style feedback (room F1, door F1, top failures).</li><li><b>Practice and held-out sets.</b> 24 public practice plans give the feedback; 64 held-out plans from other buildings give the final score and never feed back.</li><li><b>Runnable.</b> Docker files for the sandbox and the grader are in the dataset (<code>task/environment</code>, <code>task/tests</code>).</li><li><b>Benchmark evidence.</b> The Beam and Kimi K3 runs below show what the environment measures.</li></ul><p>We did not update Beam's weights; in these runs Beam improves only in-context, by retrying with the environment's feedback. The environment is ready for RL fine-tuning of Beam (open weights).</p>
</details>
<details><summary><b>Results</b></summary><table><tr><th>Model</th><th>Rounds</th><th>Best public strict</th><th>Best public partial</th><th>Held-out strict (64 plans)</th><th>Held-out partial</th><th>Best round</th></tr>
<tr><td>Beam low effort, 3 rounds (repeat 1)</td><td>3</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>round 1</td></tr>
<tr><td>Beam low effort, 3 rounds (repeat 2)</td><td>3</td><td>0.000</td><td>0.026</td><td>0.000</td><td>0.035</td><td>round 3</td></tr>
<tr><td>Beam low effort, 3 rounds (repeat 3)</td><td>3</td><td>0.000</td><td>0.058</td><td>0.000</td><td>0.061</td><td>round 2</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 1)</td><td>3</td><td>0.000</td><td>0.101</td><td>0.000</td><td>0.102</td><td>round 1</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 2)</td><td>3</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>round 1</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 3)</td><td>3</td><td>0.000</td><td>0.136</td><td>0.000</td><td>0.123</td><td>round 1</td></tr>
<tr><td>Beam high effort, 3 rounds</td><td>3</td><td>0.000</td><td>0.010</td><td>0.000</td><td>0.016</td><td>round 2</td></tr>
<tr><td>Beam high + playbook, 3 rounds</td><td>3</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>round 1</td></tr>
<tr><td>Beam low, 10 rounds, each revises the best code so far</td><td>10</td><td>0.000</td><td>0.182</td><td>0.000</td><td>0.179</td><td>round 9</td></tr>
<tr><td>Beam high, 5 rounds, best of 4 samples per round</td><td>5</td><td>0.000</td><td>0.181</td><td>0.000</td><td>0.176</td><td>round 4</td></tr>
<tr><td colspan="7" style="text-align:left">Kimi K3: no reply to round 1 in 31 min; did not finish.</td></tr>
<tr><td>Earlier baseline: Opus 5.5, one 2-hour agent run with tools (a different setup; strict score)</td><td>-</td><td>-</td><td>-</td><td>0.032</td><td>-</td><td>-</td></tr>
<tr><td>Reference solution (hand-written, for scale)</td><td>-</td><td>-</td><td>-</td><td>0.242</td><td>0.586</td><td>-</td></tr></table>
<p class="note"><b>Strict</b> = the official score: plan score = room F1 x door F1 (rooms matched one to one at IoU ≥ 0.7, a door counts when its rooms match and its centre is within 0.5 m); gate g = min(1, mean clean / 0.6); strict = g x mean over the 48 hard plans. <b>Partial</b> = mean over plans of (room F1 + door F1) / 2, the dense RL signal. Public scores drive the feedback; held-out plans are used only for the final score.</p>
</details>
<details><summary><b>Held-out results per style (64 plans, 16 per style)</b></summary><table><tr><th>Run</th><th>Room F1 clean</th><th>outline+clutter</th><th>clutter+scan</th><th>outline+scan</th><th>Door F1 (mean)</th><th>Strict</th><th>Partial</th></tr>
<tr><td>Beam low effort, 3 rounds (repeat 1)</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr>
<tr><td>Beam low effort, 3 rounds (repeat 2)</td><td>0.045</td><td>0.028</td><td>0.000</td><td>0.203</td><td>0.000</td><td>0.000</td><td>0.035</td></tr>
<tr><td>Beam low effort, 3 rounds (repeat 3)</td><td>0.149</td><td>0.011</td><td>0.036</td><td>0.291</td><td>0.000</td><td>0.000</td><td>0.061</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 1)</td><td>0.695</td><td>0.025</td><td>0.013</td><td>0.087</td><td>0.000</td><td>0.000</td><td>0.102</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 2)</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 3)</td><td>0.813</td><td>0.035</td><td>0.013</td><td>0.105</td><td>0.005</td><td>0.000</td><td>0.123</td></tr>
<tr><td>Beam high effort, 3 rounds</td><td>0.059</td><td>0.024</td><td>0.024</td><td>0.018</td><td>0.000</td><td>0.000</td><td>0.016</td></tr>
<tr><td>Beam high + playbook, 3 rounds</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr>
<tr><td>Beam low, 10 rounds, each revises the best code so far</td><td>0.803</td><td>0.043</td><td>0.036</td><td>0.550</td><td>0.000</td><td>0.000</td><td>0.179</td></tr>
<tr><td>Beam high, 5 rounds, best of 4 samples per round</td><td>0.789</td><td>0.010</td><td>0.050</td><td>0.560</td><td>0.000</td><td>0.000</td><td>0.176</td></tr>
<tr><td>Reference solution</td><td>0.969</td><td>0.524</td><td>0.779</td><td>0.651</td><td>0.441</td><td>0.242</td><td>0.586</td></tr>
</table>
</details>
<details><summary><b>Does a playbook in the prompt help? (held-out, 3 repeats each)</b></summary><p>The "playbook" is a 46-line list of general computer-vision techniques added to the prompt (<code>code/playbook.md</code>); no answers, no plan-specific hints. With tips the mean is about 2.4x higher, but n = 3 and one repeat in each group scored 0, so this is not a firm result.</p><table><tr><th>Group</th><th>Runs</th><th>Held-out strict</th><th>Held-out partial</th></tr>
<tr><td>beam-low (no playbook)</td><td>3</td><td>mean 0.000, spread 0.000 to 0.000 (runs: 0.000, 0.000, 0.000)</td><td>mean 0.032, spread 0.000 to 0.061 (runs: 0.000, 0.035, 0.061)</td></tr>
<tr><td>beam-low + playbook</td><td>3</td><td>mean 0.000, spread 0.000 to 0.000 (runs: 0.000, 0.000, 0.000)</td><td>mean 0.075, spread 0.000 to 0.123 (runs: 0.102, 0.000, 0.123)</td></tr>
</table>
</details>
<details><summary><b>Score per round (public plans)</b></summary><svg viewBox="0 0 900 340" width="100%" style="max-width:900px" xmlns="http://www.w3.org/2000/svg" font-family="system-ui" font-size="12">
<line x1="55" x2="560" y1="295.0" y2="295.0" stroke="#ddd"/><text x="49" y="299.0" text-anchor="end">0.00</text>
<line x1="55" x2="560" y1="240.0" y2="240.0" stroke="#ddd"/><text x="49" y="244.0" text-anchor="end">0.04</text>
<line x1="55" x2="560" y1="185.0" y2="185.0" stroke="#ddd"/><text x="49" y="189.0" text-anchor="end">0.08</text>
<line x1="55" x2="560" y1="130.0" y2="130.0" stroke="#ddd"/><text x="49" y="134.0" text-anchor="end">0.13</text>
<line x1="55" x2="560" y1="75.0" y2="75.0" stroke="#ddd"/><text x="49" y="79.0" text-anchor="end">0.17</text>
<line x1="55" x2="560" y1="20.0" y2="20.0" stroke="#ddd"/><text x="49" y="24.0" text-anchor="end">0.21</text>
<text x="55.0" y="313" text-anchor="middle">1</text>
<text x="111.1" y="313" text-anchor="middle">2</text>
<text x="167.2" y="313" text-anchor="middle">3</text>
<text x="223.3" y="313" text-anchor="middle">4</text>
<text x="279.4" y="313" text-anchor="middle">5</text>
<text x="335.6" y="313" text-anchor="middle">6</text>
<text x="391.7" y="313" text-anchor="middle">7</text>
<text x="447.8" y="313" text-anchor="middle">8</text>
<text x="503.9" y="313" text-anchor="middle">9</text>
<text x="560.0" y="313" text-anchor="middle">10</text>
<text x="307.5" y="334" text-anchor="middle">round</text>
<text x="14" y="147.5" transform="rotate(-90 14 147.5)" text-anchor="middle">public partial credit</text>
<polyline fill="none" stroke="#2b8cbe" stroke-width="2.5" points="55.0,295.0 111.1,295.0 167.2,295.0"/>
<circle cx="55.0" cy="295.0" r="4" fill="#2b8cbe"><title>Beam low effort, 3 rounds (repeat 1) round 1: partial 0.000, strict 0.000</title></circle>
<circle cx="111.1" cy="295.0" r="4" fill="#2b8cbe"><title>Beam low effort, 3 rounds (repeat 1) round 2: partial 0.000, strict 0.000</title></circle>
<circle cx="167.2" cy="295.0" r="4" fill="#2b8cbe"><title>Beam low effort, 3 rounds (repeat 1) round 3: partial 0.000, strict 0.000</title></circle>
<line x1="572" x2="592" y1="30" y2="30" stroke="#2b8cbe" stroke-width="3"/><text x="598" y="34">Beam low</text>
<polyline fill="none" stroke="#41ab5d" stroke-width="2.5" points="55.0,163.1 111.1,295.0 167.2,295.0"/>
<circle cx="55.0" cy="163.1" r="4" fill="#41ab5d"><title>Beam low + playbook, 3 rounds (repeat 1) round 1: partial 0.101, strict 0.000</title></circle>
<circle cx="111.1" cy="295.0" r="4" fill="#41ab5d"><title>Beam low + playbook, 3 rounds (repeat 1) round 2: partial 0.000, strict 0.000</title></circle>
<circle cx="167.2" cy="295.0" r="4" fill="#41ab5d"><title>Beam low + playbook, 3 rounds (repeat 1) round 3: partial 0.000, strict 0.000</title></circle>
<line x1="572" x2="592" y1="50" y2="50" stroke="#41ab5d" stroke-width="3"/><text x="598" y="54">Beam low + playbook</text>
<polyline fill="none" stroke="#08306b" stroke-width="2.5" points="55.0,282.0 111.1,281.4 167.2,289.5"/>
<circle cx="55.0" cy="282.0" r="4" fill="#08306b"><title>Beam high effort, 3 rounds round 1: partial 0.010, strict 0.000</title></circle>
<circle cx="111.1" cy="281.4" r="4" fill="#08306b"><title>Beam high effort, 3 rounds round 2: partial 0.010, strict 0.000</title></circle>
<circle cx="167.2" cy="289.5" r="4" fill="#08306b"><title>Beam high effort, 3 rounds round 3: partial 0.004, strict 0.000</title></circle>
<line x1="572" x2="592" y1="70" y2="70" stroke="#08306b" stroke-width="3"/><text x="598" y="74">Beam high</text>
<polyline fill="none" stroke="#756bb1" stroke-width="2.5" points="55.0,295.0 111.1,295.0 167.2,295.0"/>
<circle cx="55.0" cy="295.0" r="4" fill="#756bb1"><title>Beam high + playbook, 3 rounds round 1: partial 0.000, strict 0.000</title></circle>
<circle cx="111.1" cy="295.0" r="4" fill="#756bb1"><title>Beam high + playbook, 3 rounds round 2: partial 0.000, strict 0.000</title></circle>
<circle cx="167.2" cy="295.0" r="4" fill="#756bb1"><title>Beam high + playbook, 3 rounds round 3: partial 0.000, strict 0.000</title></circle>
<line x1="572" x2="592" y1="90" y2="90" stroke="#756bb1" stroke-width="3"/><text x="598" y="94">Beam high + playbook</text>
<polyline fill="none" stroke="#d6604d" stroke-width="2.5" points="55.0,295.0 111.1,262.1 167.2,272.1 223.3,265.9 279.4,60.3 335.6,192.7 391.7,66.5 447.8,64.9 503.9,55.9 560.0,58.8"/>
<circle cx="55.0" cy="295.0" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 1: partial 0.000, strict 0.000</title></circle>
<circle cx="111.1" cy="262.1" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 2: partial 0.025, strict 0.000</title></circle>
<circle cx="167.2" cy="272.1" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 3: partial 0.018, strict 0.000</title></circle>
<circle cx="223.3" cy="265.9" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 4: partial 0.022, strict 0.000</title></circle>
<circle cx="279.4" cy="60.3" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 5: partial 0.179, strict 0.000</title></circle>
<circle cx="335.6" cy="192.7" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 6: partial 0.078, strict 0.000</title></circle>
<circle cx="391.7" cy="66.5" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 7: partial 0.174, strict 0.000</title></circle>
<circle cx="447.8" cy="64.9" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 8: partial 0.176, strict 0.000</title></circle>
<circle cx="503.9" cy="55.9" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 9: partial 0.182, strict 0.000</title></circle>
<circle cx="560.0" cy="58.8" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 10: partial 0.180, strict 0.000</title></circle>
<line x1="572" x2="592" y1="110" y2="110" stroke="#d6604d" stroke-width="3"/><text x="598" y="114">Beam low, best-so-far, 10 rounds</text>
<polyline fill="none" stroke="#636363" stroke-width="2.5" points="55.0,92.4 111.1,111.0 167.2,173.0 223.3,58.0 279.4,65.7"/>
<circle cx="55.0" cy="92.4" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 1: partial 0.155, strict 0.000</title></circle>
<circle cx="111.1" cy="111.0" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 2: partial 0.140, strict 0.000</title></circle>
<circle cx="167.2" cy="173.0" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 3: partial 0.093, strict 0.000</title></circle>
<circle cx="223.3" cy="58.0" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 4: partial 0.181, strict 0.000</title></circle>
<circle cx="279.4" cy="65.7" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 5: partial 0.175, strict 0.000</title></circle>
<line x1="572" x2="592" y1="130" y2="130" stroke="#636363" stroke-width="3"/><text x="598" y="134">Beam high, best of 4, 5 rounds</text>
</svg>
</details>
<details><summary><b>How Beam used the feedback (Beam low, 10 rounds, each revises the best code so far)</b></summary><p>Each round Beam gets its previous program and the grader's report (per-style room F1 / door F1, top failures from the public plans) and writes a new <code>run.py</code>. Below: Beam's own reasoning (<code>reasoning_content</code>) in the rounds where its score rose, with the score before and after.</p>
<table><tr><th>Round</th><th>Public partial</th><th>Public strict</th><th>Room F1 clean / outline+clutter / clutter+scan / outline+scan</th><th>Lines of code</th><th>Output tokens (reasoning)</th><th>Files</th></tr>
<tr><td>1</td><td>0.000</td><td>0.000</td><td>0.000 / 0.000 / 0.000 / 0.000</td><td>109</td><td>1696 (577)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-1/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-1/reasoning.md">reasoning</a></td></tr>
<tr><td>2</td><td>0.025</td><td>0.000</td><td>0.151 / 0.050 / 0.000 / 0.000</td><td>101</td><td>1136 (154)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-2/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-2/reasoning.md">reasoning</a></td></tr>
<tr><td>3</td><td>0.018</td><td>0.000</td><td>0.118 / 0.022 / 0.000 / 0.000</td><td>101</td><td>1701 (710)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-3/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-3/reasoning.md">reasoning</a></td></tr>
<tr><td>4</td><td>0.022</td><td>0.000</td><td>0.000 / 0.000 / 0.000 / 0.178</td><td>105</td><td>1312 (229)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-4/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-4/reasoning.md">reasoning</a></td></tr>
<tr><td>5</td><td>0.179</td><td>0.000</td><td>0.707 / 0.117 / 0.123 / 0.486</td><td>107</td><td>1216 (152)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-5/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-5/reasoning.md">reasoning</a></td></tr>
<tr><td>6</td><td>0.078</td><td>0.000</td><td>0.373 / 0.085 / 0.125 / 0.042</td><td>97</td><td>1058 (103)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-6/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-6/reasoning.md">reasoning</a></td></tr>
<tr><td>7</td><td>0.174</td><td>0.000</td><td>0.739 / 0.110 / 0.088 / 0.458</td><td>97</td><td>1173 (171)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-7/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-7/reasoning.md">reasoning</a></td></tr>
<tr><td>8</td><td>0.176</td><td>0.000</td><td>0.708 / 0.116 / 0.122 / 0.459</td><td>160</td><td>2329 (696)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-8/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-8/reasoning.md">reasoning</a></td></tr>
<tr><td>9</td><td>0.182</td><td>0.000</td><td>0.739 / 0.116 / 0.107 / 0.497</td><td>128</td><td>2067 (708)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-9/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-9/reasoning.md">reasoning</a></td></tr>
<tr><td>10</td><td>0.180</td><td>0.000</td><td>0.739 / 0.116 / 0.097 / 0.491</td><td>168</td><td>1911 (74)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-10/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-10/reasoning.md">reasoning</a></td></tr>
</table>
<blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em;color:#333"><b>Round 2</b> (public partial 0.000 → 0.025): “The previous submission failed because it wrote no rooms? Actually it says pred 0 rooms. Probably adaptive threshold or contour issue due to image dimensions mismatch? cv2.imread on grayscale 8-bit should work. But maybe plan.png is larger? We resized. The wall mask maybe all white? Let's not overthink; output empty likely due to exception? But status ok. Could be min_area too high? No. Given time budget, I'll provide a slightly improved but stil ...”</blockquote>
<blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em;color:#333"><b>Round 5</b> (public partial 0.022 → 0.179): “My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach. Let me think about what's needed: detect walls, close small gaps, find free regions. The previous code closed gaps with morphology but maybe threshold was wrong, or morphology too aggressive. Given the complexity and time, I'll improve: use Otsu threshold, invert properly, handle scan styles with adaptive thresholding. A ...”</blockquote>
<blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em;color:#333"><b>Round 7</b> (public partial 0.078 → 0.174): “The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet. Key: rooms IoU low. Need better thresholding, keep wall gaps. Let me write improved version using Otsu + closing small gaps under 0.5m only, proper free space. Also handle scan rotation? The polygons move with image, so no need to unrotate. Let me improve: use adaptive threshol ...”</blockquote>
</details>
<details><summary><b>What Beam's program sees, including where it still fails (p06: a rotated, noisy scan)</b></summary><p>Left: the rooms (colour), doors (red dot; green = door to outside) and door edges recovered by Beam low, 10 rounds, each revises the best code so far (round 9), drawn over the plan. Right: the true room graph. The blue line is a <b>wheelchair route to the exit</b>: a networkx shortest path over the door graph from the room farthest from the exit to the outside door, computed only from the recovered map. Blue dot = start.</p>
<h3>p17 (clean)</h3><div class="figs"><div><img src="img/p17-beam.png" alt="Beam recovered graph for p17"><div>Beam</div></div><div><img src="img/p17-truth.png" alt="true graph for p17"><div>Truth</div></div></div>
<div class="cap">Rooms matched 4/4 (pred 5), doors matched 0/4 (pred 0), plan score 0.000, partial 0.444. Beam map: no route: no recovered room has a door path to outside. True route: 2 door(s), narrowest 1.04 m, all at least 0.8 m wide.</div>
<h3>p01 (outline+clutter)</h3><div class="figs"><div><img src="img/p01-beam.png" alt="Beam recovered graph for p01"><div>Beam</div></div><div><img src="img/p01-truth.png" alt="true graph for p01"><div>Truth</div></div></div>
<div class="cap">Rooms matched 3/5 (pred 10), doors matched 0/5 (pred 0), plan score 0.000, partial 0.200. Beam map: no route: no recovered room has a door path to outside. True route: 2 door(s), narrowest 0.84 m, all at least 0.8 m wide.</div>
<h3>p06 (clutter+scan)</h3><div class="figs"><div><img src="img/p06-beam.png" alt="Beam recovered graph for p06"><div>Beam</div></div><div><img src="img/p06-truth.png" alt="true graph for p06"><div>Truth</div></div></div>
<div class="cap">Rooms matched 2/5 (pred 10), doors matched 0/5 (pred 6), plan score 0.000, partial 0.133. Beam map: route found through 1 door(s); 1 of 10 recovered rooms reach the exit. True route: 3 door(s), narrowest 0.68 m, 1 under 0.8 m (too narrow for many wheelchairs).</div>
<h3>p12 (outline+scan)</h3><div class="figs"><div><img src="img/p12-beam.png" alt="Beam recovered graph for p12"><div>Beam</div></div><div><img src="img/p12-truth.png" alt="true graph for p12"><div>Truth</div></div></div>
<div class="cap">Rooms matched 2/4 (pred 2), doors matched 0/4 (pred 0), plan score 0.000, partial 0.333. Beam map: no route: no recovered room has a door path to outside. True route: 2 door(s), narrowest 0.80 m, 1 under 0.8 m (too narrow for many wheelchairs).</div>
</details>
<details><summary><b>Why this is useful</b></summary><p>Floor plans exist mostly as images. A room graph turns one into something a program can answer questions about: is every room reachable from the exit, what is the step-free route for a wheelchair, how many bedrooms, which rooms are interior. The environment is a ready-made, exactly graded RL task for teaching a text-only model to do visual work through code, with dense partial credit so early programs still get a learning signal.</p>
</details>
<details><summary><b>Credit</b></summary><p>Plan geometry: Swiss Dwellings v3.0.0, Zenodo, DOI <a href="https://doi.org/10.5281/zenodo.7788422">10.5281/zenodo.7788422</a>, CC BY 4.0. Plans re-drawn from that geometry. Built by Hamood (hamoodey) for Reflection's Build with Beam contest. Beam: <code>Beam-1.0-501B-A23B</code> via the Reflection API.</p></details>
<details><summary><b>Can the reward be gamed?</b></summary><ul>
<li>Held-out plans come from 64 buildings that appear in no practice plan; feedback in the loop uses only the 24 practice plans.</li>
<li>The program runs as an unprivileged user that owns no file, offline, with 30 s CPU per plan; it sees only <code>plan.png</code> and <code>manifest.json</code>.</li>
<li>Output is clamped: at most 2 MB, 100 rooms, 300 doors, 50,000 points; any schema error, or rooms overlapping by more than 10%, scores the plan 0.</li>
<li>Rooms are matched one to one (Hungarian) at IoU 0.7, so extra or split rooms lower F1; a door counts only if both its rooms match and its centre is within 0.5 m.</li>
<li>Spamming doors fails: a clean plan with door precision under 0.5 counts as 0 in the gate.</li>
<li>Sanity checks: the empty stub scores 0.000; the hand-written reference scores 0.242 strict.</li></ul></details>
</body></html> |