floorplan-graph-rl / index.html
hamoodey's picture
update
baf8832 verified
Raw History Blame Contribute Delete
31.5 kB
<!doctype html><html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Floorplan Graph RL: Beam reads real floor plans by writing code</title><style>body{font-family:system-ui,-apple-system,sans-serif;max-width:1100px;margin:2em auto;padding:0 1em;line-height:1.5;color:#222}table{border-collapse:collapse;margin:1em 0}td,th{border:1px solid #ccc;padding:6px 10px;text-align:right}td:first-child,th:first-child{text-align:left}.figs{display:grid;grid-template-columns:1fr 1fr;gap:12px}.figs img{width:100%;border:1px solid #ddd}.cap{font-size:.9em;color:#444;margin-bottom:1.5em}code{background:#f3f3f3;padding:1px 4px}.note{color:#666;font-size:.9em}details{margin:.6em 0;border:1px solid #e3e3e3;border-radius:6px;padding:.4em .8em}summary{cursor:pointer;font-size:1.1em}</style></head><body>
<h1>Floorplan Graph RL: Beam reads real apartment floor plans by writing code</h1>
<p style='font-size:1.3em'><b>Beam can't see an image. So it learned to write its own eyes.</b></p>
<p style='font-size:1.15em'><b>A text-only Beam agent writes its own computer-vision pipeline, learns from exact grader feedback, and improves without weight updates.</b> Its best 10-round run reaches <b>0.179 held-out partial credit</b> on 64 unseen real apartment plans and <b>0.803 room F1 on clean plans</b>.</p><p><b>Beam actually learned:</b> after the grader reported zero rooms on scans, Beam diagnosed its thresholding failure and adopted adaptive thresholding. Public partial credit jumped from <b>0.022 to 0.179</b>.</p><p><b>The problem.</b> Many floor plans exist only as images or scans. A room graph (rooms + doors) answers "can a wheelchair reach every room?", "how do I get out?" and "how many rooms?". This environment teaches a text-only model to build that graph from a real plan, and grades it exactly.</p><p><b>In one screen.</b></p>
<ul>
<li><b>A novel Beam loop:</b> Beam writes a complete vision program, the environment executes it on real plans, and Beam uses exact rewards and failure reports to revise it. Every prompt, reasoning trace, program, and score is published.</li>
<li><b>Measured improvement:</b> the 10-round run rises from <b>0.000 to 0.182 public partial credit</b>; the selected round scores <b>0.179 on 64 held-out plans from other buildings</b>. A 46-line general tips playbook also lifts Beam low from <b>0.032 to 0.075</b> held-out (3 repeats each; wide spread).</li>
<li><b>A reusable, unsaturated RL environment:</b> <code>reset()</code> / <code>step(code)</code>, strict and dense rewards, 24 practice + 64 held-out plans, Docker grading, and a GRPO example. Strict reward remains <b>0.000</b> versus <b>0.242</b> for the reference: rooms are emerging; doors are the open research problem.</li>
</ul><p>Data, grader, Docker files and every run: <a href='https://huggingface.co/datasets/hamoodey/floorplan-graph-rl'>huggingface.co/datasets/hamoodey/floorplan-graph-rl</a></p><p><img src="img/hero.png" style="width:100%" alt="input plan, true room graph, Beam graph, route to exit"></p><p class="note">Plan p17 (clean style). Panel 3: rooms recovered by Beam's own program (Beam low, round 9). Panel 4: the step-free route to the exit, a networkx shortest path over the true door graph; this is what a recovered graph is for. Beam's doors are not right yet (door F1 0), so its own route would not be trustworthy.</p><table><tr><th>Evidence</th><th>Result</th></tr><tr><td>Beam low, 10 feedback rounds</td><td><b>0.000 &rarr; 0.182 public partial</b>; <b>0.179 held-out partial</b></td></tr><tr><td>General tips playbook, 3 repeats each</td><td><b>0.032 &rarr; 0.075 held-out partial</b></td></tr><tr><td>Clean-plan room recovery, best 10-round run</td><td><b>0.803 room F1</b></td></tr><tr><td>Strict held-out reward</td><td><b>0.000</b> for every Beam setup; <b>0.242</b> reference</td></tr></table><div style="display:flex;gap:16px;flex-wrap:wrap;align-items:flex-start"><img src="img/rounds.gif" style="max-width:460px;width:100%;border:1px solid #ddd" alt="Beam rooms per round"><div style="flex:1;min-width:300px"><img src="img/bars.png" style="width:100%" alt="held-out partial credit by setup"></div></div><p><img src="img/rounds-chart.png" style="width:100%" alt="score per round"></p><h2>Use it for RL</h2><pre style="background:#f6f6f6;padding:1em;overflow-x:auto"><code>from code.env import FloorplanEnv, extract_code
env = FloorplanEnv(set="public") # 24 practice plans; "hidden" only for the final score
prompt = env.reset() # instruction + contract + style guide
reward, partial, feedback, info = env.step(extract_code(model_reply))
next_prompt = prompt + previous_code + feedback # the in-context loop used here
def floorplan_reward(completions, **kw): # GRPO-style reward (e.g. TRL GRPOTrainer)
return [sum(FloorplanEnv("public").step(extract_code(c if isinstance(c, str) else c[-1]["content"]))[:2])
for c in completions] # strict + dense partial credit</code></pre><p>Unsaturated: strict reward is still 0.000 for every model loop here, while the hand-written reference reaches 0.242. Lots of room to train.</p>
<h2>What generalizes for anyone using Beam on code-writing tasks</h2>
<ul>
<li>Feedback beats a single try: graded retries took Beam from 0.000 (round 1) to 0.179 held-out partial credit with no weight change.</li>
<li>Write down what worked: a 46-line tips playbook more than doubled the held-out score (0.032 &rarr; 0.075, 3 repeats each).</li>
<li>Grade on data the feedback never saw: 64 held-out plans from other buildings keep the gain honest.</li>
<li>Score partial credit as well as strict: strict was 0.000 in every run, so only partial credit shows learning.</li>
</ul><h2>Beam reading its failure report (<code>reasoning_content</code>, Beam low, 10-round run)</h2><blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em"><b>Round 5</b> (practice-plan partial credit 0.022 &rarr; 0.179): &ldquo;My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach.&rdquo;</blockquote><blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em"><b>Round 7</b> (practice-plan partial credit 0.078 &rarr; 0.174): &ldquo;The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet.&rdquo;</blockquote><h2>Why trust these numbers</h2>
<p>The grader is exact (rooms at IoU 0.7, doors within 0.5 m). Feedback uses only the 24 practice plans; the 64 held-out plans are from other buildings and never feed back. Every prompt, reasoning trace, program and score is in <code>runs/</code>.</p>
<p><b>Known limits.</b> Kimi K3 did not finish (no reply to its round-1 request in 31 min); Beam high effort finished (held-out 0.016 in a single 3-round run; 0.176 with best of 4 samples per round). Most numbers are one run; the playbook result has 3 repeats with a wide spread. Doors are nearly unsolved (door F1 0.000 on every setup but one). We did not update Beam's weights.</p><details><summary><b>What this entry is</b></summary><ul><li><b>An RL environment for Beam.</b> <code>FloorplanEnv.reset()</code> gives the task prompt; <code>step(code)</code> runs Beam's <code>run.py</code> and returns the strict reward, a dense partial-credit reward and per-style feedback (room F1, door F1, top failures).</li><li><b>Practice and held-out sets.</b> 24 public practice plans give the feedback; 64 held-out plans from other buildings give the final score and never feed back.</li><li><b>Runnable.</b> Docker files for the sandbox and the grader are in the dataset (<code>task/environment</code>, <code>task/tests</code>).</li><li><b>Benchmark evidence.</b> The Beam and Kimi K3 runs below show what the environment measures.</li></ul><p>We did not update Beam's weights; in these runs Beam improves only in-context, by retrying with the environment's feedback. The environment is ready for RL fine-tuning of Beam (open weights).</p>
</details>
<details><summary><b>Results</b></summary><table><tr><th>Model</th><th>Rounds</th><th>Best public strict</th><th>Best public partial</th><th>Held-out strict (64 plans)</th><th>Held-out partial</th><th>Best round</th></tr>
<tr><td>Beam low effort, 3 rounds (repeat 1)</td><td>3</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>round 1</td></tr>
<tr><td>Beam low effort, 3 rounds (repeat 2)</td><td>3</td><td>0.000</td><td>0.026</td><td>0.000</td><td>0.035</td><td>round 3</td></tr>
<tr><td>Beam low effort, 3 rounds (repeat 3)</td><td>3</td><td>0.000</td><td>0.058</td><td>0.000</td><td>0.061</td><td>round 2</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 1)</td><td>3</td><td>0.000</td><td>0.101</td><td>0.000</td><td>0.102</td><td>round 1</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 2)</td><td>3</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>round 1</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 3)</td><td>3</td><td>0.000</td><td>0.136</td><td>0.000</td><td>0.123</td><td>round 1</td></tr>
<tr><td>Beam high effort, 3 rounds</td><td>3</td><td>0.000</td><td>0.010</td><td>0.000</td><td>0.016</td><td>round 2</td></tr>
<tr><td>Beam high + playbook, 3 rounds</td><td>3</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>round 1</td></tr>
<tr><td>Beam low, 10 rounds, each revises the best code so far</td><td>10</td><td>0.000</td><td>0.182</td><td>0.000</td><td>0.179</td><td>round 9</td></tr>
<tr><td>Beam high, 5 rounds, best of 4 samples per round</td><td>5</td><td>0.000</td><td>0.181</td><td>0.000</td><td>0.176</td><td>round 4</td></tr>
<tr><td colspan="7" style="text-align:left">Kimi K3: no reply to round 1 in 31 min; did not finish.</td></tr>
<tr><td>Earlier baseline: Opus 5.5, one 2-hour agent run with tools (a different setup; strict score)</td><td>-</td><td>-</td><td>-</td><td>0.032</td><td>-</td><td>-</td></tr>
<tr><td>Reference solution (hand-written, for scale)</td><td>-</td><td>-</td><td>-</td><td>0.242</td><td>0.586</td><td>-</td></tr></table>
<p class="note"><b>Strict</b> = the official score: plan score = room F1 x door F1 (rooms matched one to one at IoU &ge; 0.7, a door counts when its rooms match and its centre is within 0.5 m); gate g = min(1, mean clean / 0.6); strict = g x mean over the 48 hard plans. <b>Partial</b> = mean over plans of (room F1 + door F1) / 2, the dense RL signal. Public scores drive the feedback; held-out plans are used only for the final score.</p>
</details>
<details><summary><b>Held-out results per style (64 plans, 16 per style)</b></summary><table><tr><th>Run</th><th>Room F1 clean</th><th>outline+clutter</th><th>clutter+scan</th><th>outline+scan</th><th>Door F1 (mean)</th><th>Strict</th><th>Partial</th></tr>
<tr><td>Beam low effort, 3 rounds (repeat 1)</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr>
<tr><td>Beam low effort, 3 rounds (repeat 2)</td><td>0.045</td><td>0.028</td><td>0.000</td><td>0.203</td><td>0.000</td><td>0.000</td><td>0.035</td></tr>
<tr><td>Beam low effort, 3 rounds (repeat 3)</td><td>0.149</td><td>0.011</td><td>0.036</td><td>0.291</td><td>0.000</td><td>0.000</td><td>0.061</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 1)</td><td>0.695</td><td>0.025</td><td>0.013</td><td>0.087</td><td>0.000</td><td>0.000</td><td>0.102</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 2)</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr>
<tr><td>Beam low + playbook, 3 rounds (repeat 3)</td><td>0.813</td><td>0.035</td><td>0.013</td><td>0.105</td><td>0.005</td><td>0.000</td><td>0.123</td></tr>
<tr><td>Beam high effort, 3 rounds</td><td>0.059</td><td>0.024</td><td>0.024</td><td>0.018</td><td>0.000</td><td>0.000</td><td>0.016</td></tr>
<tr><td>Beam high + playbook, 3 rounds</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr>
<tr><td>Beam low, 10 rounds, each revises the best code so far</td><td>0.803</td><td>0.043</td><td>0.036</td><td>0.550</td><td>0.000</td><td>0.000</td><td>0.179</td></tr>
<tr><td>Beam high, 5 rounds, best of 4 samples per round</td><td>0.789</td><td>0.010</td><td>0.050</td><td>0.560</td><td>0.000</td><td>0.000</td><td>0.176</td></tr>
<tr><td>Reference solution</td><td>0.969</td><td>0.524</td><td>0.779</td><td>0.651</td><td>0.441</td><td>0.242</td><td>0.586</td></tr>
</table>
</details>
<details><summary><b>Does a playbook in the prompt help? (held-out, 3 repeats each)</b></summary><p>The "playbook" is a 46-line list of general computer-vision techniques added to the prompt (<code>code/playbook.md</code>); no answers, no plan-specific hints. With tips the mean is about 2.4x higher, but n = 3 and one repeat in each group scored 0, so this is not a firm result.</p><table><tr><th>Group</th><th>Runs</th><th>Held-out strict</th><th>Held-out partial</th></tr>
<tr><td>beam-low (no playbook)</td><td>3</td><td>mean 0.000, spread 0.000 to 0.000 (runs: 0.000, 0.000, 0.000)</td><td>mean 0.032, spread 0.000 to 0.061 (runs: 0.000, 0.035, 0.061)</td></tr>
<tr><td>beam-low + playbook</td><td>3</td><td>mean 0.000, spread 0.000 to 0.000 (runs: 0.000, 0.000, 0.000)</td><td>mean 0.075, spread 0.000 to 0.123 (runs: 0.102, 0.000, 0.123)</td></tr>
</table>
</details>
<details><summary><b>Score per round (public plans)</b></summary><svg viewBox="0 0 900 340" width="100%" style="max-width:900px" xmlns="http://www.w3.org/2000/svg" font-family="system-ui" font-size="12">
<line x1="55" x2="560" y1="295.0" y2="295.0" stroke="#ddd"/><text x="49" y="299.0" text-anchor="end">0.00</text>
<line x1="55" x2="560" y1="240.0" y2="240.0" stroke="#ddd"/><text x="49" y="244.0" text-anchor="end">0.04</text>
<line x1="55" x2="560" y1="185.0" y2="185.0" stroke="#ddd"/><text x="49" y="189.0" text-anchor="end">0.08</text>
<line x1="55" x2="560" y1="130.0" y2="130.0" stroke="#ddd"/><text x="49" y="134.0" text-anchor="end">0.13</text>
<line x1="55" x2="560" y1="75.0" y2="75.0" stroke="#ddd"/><text x="49" y="79.0" text-anchor="end">0.17</text>
<line x1="55" x2="560" y1="20.0" y2="20.0" stroke="#ddd"/><text x="49" y="24.0" text-anchor="end">0.21</text>
<text x="55.0" y="313" text-anchor="middle">1</text>
<text x="111.1" y="313" text-anchor="middle">2</text>
<text x="167.2" y="313" text-anchor="middle">3</text>
<text x="223.3" y="313" text-anchor="middle">4</text>
<text x="279.4" y="313" text-anchor="middle">5</text>
<text x="335.6" y="313" text-anchor="middle">6</text>
<text x="391.7" y="313" text-anchor="middle">7</text>
<text x="447.8" y="313" text-anchor="middle">8</text>
<text x="503.9" y="313" text-anchor="middle">9</text>
<text x="560.0" y="313" text-anchor="middle">10</text>
<text x="307.5" y="334" text-anchor="middle">round</text>
<text x="14" y="147.5" transform="rotate(-90 14 147.5)" text-anchor="middle">public partial credit</text>
<polyline fill="none" stroke="#2b8cbe" stroke-width="2.5" points="55.0,295.0 111.1,295.0 167.2,295.0"/>
<circle cx="55.0" cy="295.0" r="4" fill="#2b8cbe"><title>Beam low effort, 3 rounds (repeat 1) round 1: partial 0.000, strict 0.000</title></circle>
<circle cx="111.1" cy="295.0" r="4" fill="#2b8cbe"><title>Beam low effort, 3 rounds (repeat 1) round 2: partial 0.000, strict 0.000</title></circle>
<circle cx="167.2" cy="295.0" r="4" fill="#2b8cbe"><title>Beam low effort, 3 rounds (repeat 1) round 3: partial 0.000, strict 0.000</title></circle>
<line x1="572" x2="592" y1="30" y2="30" stroke="#2b8cbe" stroke-width="3"/><text x="598" y="34">Beam low</text>
<polyline fill="none" stroke="#41ab5d" stroke-width="2.5" points="55.0,163.1 111.1,295.0 167.2,295.0"/>
<circle cx="55.0" cy="163.1" r="4" fill="#41ab5d"><title>Beam low + playbook, 3 rounds (repeat 1) round 1: partial 0.101, strict 0.000</title></circle>
<circle cx="111.1" cy="295.0" r="4" fill="#41ab5d"><title>Beam low + playbook, 3 rounds (repeat 1) round 2: partial 0.000, strict 0.000</title></circle>
<circle cx="167.2" cy="295.0" r="4" fill="#41ab5d"><title>Beam low + playbook, 3 rounds (repeat 1) round 3: partial 0.000, strict 0.000</title></circle>
<line x1="572" x2="592" y1="50" y2="50" stroke="#41ab5d" stroke-width="3"/><text x="598" y="54">Beam low + playbook</text>
<polyline fill="none" stroke="#08306b" stroke-width="2.5" points="55.0,282.0 111.1,281.4 167.2,289.5"/>
<circle cx="55.0" cy="282.0" r="4" fill="#08306b"><title>Beam high effort, 3 rounds round 1: partial 0.010, strict 0.000</title></circle>
<circle cx="111.1" cy="281.4" r="4" fill="#08306b"><title>Beam high effort, 3 rounds round 2: partial 0.010, strict 0.000</title></circle>
<circle cx="167.2" cy="289.5" r="4" fill="#08306b"><title>Beam high effort, 3 rounds round 3: partial 0.004, strict 0.000</title></circle>
<line x1="572" x2="592" y1="70" y2="70" stroke="#08306b" stroke-width="3"/><text x="598" y="74">Beam high</text>
<polyline fill="none" stroke="#756bb1" stroke-width="2.5" points="55.0,295.0 111.1,295.0 167.2,295.0"/>
<circle cx="55.0" cy="295.0" r="4" fill="#756bb1"><title>Beam high + playbook, 3 rounds round 1: partial 0.000, strict 0.000</title></circle>
<circle cx="111.1" cy="295.0" r="4" fill="#756bb1"><title>Beam high + playbook, 3 rounds round 2: partial 0.000, strict 0.000</title></circle>
<circle cx="167.2" cy="295.0" r="4" fill="#756bb1"><title>Beam high + playbook, 3 rounds round 3: partial 0.000, strict 0.000</title></circle>
<line x1="572" x2="592" y1="90" y2="90" stroke="#756bb1" stroke-width="3"/><text x="598" y="94">Beam high + playbook</text>
<polyline fill="none" stroke="#d6604d" stroke-width="2.5" points="55.0,295.0 111.1,262.1 167.2,272.1 223.3,265.9 279.4,60.3 335.6,192.7 391.7,66.5 447.8,64.9 503.9,55.9 560.0,58.8"/>
<circle cx="55.0" cy="295.0" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 1: partial 0.000, strict 0.000</title></circle>
<circle cx="111.1" cy="262.1" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 2: partial 0.025, strict 0.000</title></circle>
<circle cx="167.2" cy="272.1" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 3: partial 0.018, strict 0.000</title></circle>
<circle cx="223.3" cy="265.9" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 4: partial 0.022, strict 0.000</title></circle>
<circle cx="279.4" cy="60.3" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 5: partial 0.179, strict 0.000</title></circle>
<circle cx="335.6" cy="192.7" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 6: partial 0.078, strict 0.000</title></circle>
<circle cx="391.7" cy="66.5" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 7: partial 0.174, strict 0.000</title></circle>
<circle cx="447.8" cy="64.9" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 8: partial 0.176, strict 0.000</title></circle>
<circle cx="503.9" cy="55.9" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 9: partial 0.182, strict 0.000</title></circle>
<circle cx="560.0" cy="58.8" r="4" fill="#d6604d"><title>Beam low, 10 rounds, each revises the best code so far round 10: partial 0.180, strict 0.000</title></circle>
<line x1="572" x2="592" y1="110" y2="110" stroke="#d6604d" stroke-width="3"/><text x="598" y="114">Beam low, best-so-far, 10 rounds</text>
<polyline fill="none" stroke="#636363" stroke-width="2.5" points="55.0,92.4 111.1,111.0 167.2,173.0 223.3,58.0 279.4,65.7"/>
<circle cx="55.0" cy="92.4" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 1: partial 0.155, strict 0.000</title></circle>
<circle cx="111.1" cy="111.0" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 2: partial 0.140, strict 0.000</title></circle>
<circle cx="167.2" cy="173.0" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 3: partial 0.093, strict 0.000</title></circle>
<circle cx="223.3" cy="58.0" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 4: partial 0.181, strict 0.000</title></circle>
<circle cx="279.4" cy="65.7" r="4" fill="#636363"><title>Beam high, 5 rounds, best of 4 samples per round round 5: partial 0.175, strict 0.000</title></circle>
<line x1="572" x2="592" y1="130" y2="130" stroke="#636363" stroke-width="3"/><text x="598" y="134">Beam high, best of 4, 5 rounds</text>
</svg>
</details>
<details><summary><b>How Beam used the feedback (Beam low, 10 rounds, each revises the best code so far)</b></summary><p>Each round Beam gets its previous program and the grader's report (per-style room F1 / door F1, top failures from the public plans) and writes a new <code>run.py</code>. Below: Beam's own reasoning (<code>reasoning_content</code>) in the rounds where its score rose, with the score before and after.</p>
<table><tr><th>Round</th><th>Public partial</th><th>Public strict</th><th>Room F1 clean / outline+clutter / clutter+scan / outline+scan</th><th>Lines of code</th><th>Output tokens (reasoning)</th><th>Files</th></tr>
<tr><td>1</td><td>0.000</td><td>0.000</td><td>0.000 / 0.000 / 0.000 / 0.000</td><td>109</td><td>1696 (577)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-1/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-1/reasoning.md">reasoning</a></td></tr>
<tr><td>2</td><td>0.025</td><td>0.000</td><td>0.151 / 0.050 / 0.000 / 0.000</td><td>101</td><td>1136 (154)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-2/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-2/reasoning.md">reasoning</a></td></tr>
<tr><td>3</td><td>0.018</td><td>0.000</td><td>0.118 / 0.022 / 0.000 / 0.000</td><td>101</td><td>1701 (710)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-3/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-3/reasoning.md">reasoning</a></td></tr>
<tr><td>4</td><td>0.022</td><td>0.000</td><td>0.000 / 0.000 / 0.000 / 0.178</td><td>105</td><td>1312 (229)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-4/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-4/reasoning.md">reasoning</a></td></tr>
<tr><td>5</td><td>0.179</td><td>0.000</td><td>0.707 / 0.117 / 0.123 / 0.486</td><td>107</td><td>1216 (152)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-5/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-5/reasoning.md">reasoning</a></td></tr>
<tr><td>6</td><td>0.078</td><td>0.000</td><td>0.373 / 0.085 / 0.125 / 0.042</td><td>97</td><td>1058 (103)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-6/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-6/reasoning.md">reasoning</a></td></tr>
<tr><td>7</td><td>0.174</td><td>0.000</td><td>0.739 / 0.110 / 0.088 / 0.458</td><td>97</td><td>1173 (171)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-7/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-7/reasoning.md">reasoning</a></td></tr>
<tr><td>8</td><td>0.176</td><td>0.000</td><td>0.708 / 0.116 / 0.122 / 0.459</td><td>160</td><td>2329 (696)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-8/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-8/reasoning.md">reasoning</a></td></tr>
<tr><td>9</td><td>0.182</td><td>0.000</td><td>0.739 / 0.116 / 0.107 / 0.497</td><td>128</td><td>2067 (708)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-9/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-9/reasoning.md">reasoning</a></td></tr>
<tr><td>10</td><td>0.180</td><td>0.000</td><td>0.739 / 0.116 / 0.097 / 0.491</td><td>168</td><td>1911 (74)</td><td><a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-10/run.py">run.py</a> 路 <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl/blob/main/runs/beam-low-best10/round-10/reasoning.md">reasoning</a></td></tr>
</table>
<blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em;color:#333"><b>Round 2</b> (public partial 0.000 &rarr; 0.025): &ldquo;The previous submission failed because it wrote no rooms? Actually it says pred 0 rooms. Probably adaptive threshold or contour issue due to image dimensions mismatch? cv2.imread on grayscale 8-bit should work. But maybe plan.png is larger? We resized. The wall mask maybe all white? Let&#x27;s not overthink; output empty likely due to exception? But status ok. Could be min_area too high? No. Given time budget, I&#x27;ll provide a slightly improved but stil ...&rdquo;</blockquote>
<blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em;color:#333"><b>Round 5</b> (public partial 0.022 &rarr; 0.179): &ldquo;My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach. Let me think about what&#x27;s needed: detect walls, close small gaps, find free regions. The previous code closed gaps with morphology but maybe threshold was wrong, or morphology too aggressive. Given the complexity and time, I&#x27;ll improve: use Otsu threshold, invert properly, handle scan styles with adaptive thresholding. A ...&rdquo;</blockquote>
<blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em;color:#333"><b>Round 7</b> (public partial 0.078 &rarr; 0.174): &ldquo;The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet. Key: rooms IoU low. Need better thresholding, keep wall gaps. Let me write improved version using Otsu + closing small gaps under 0.5m only, proper free space. Also handle scan rotation? The polygons move with image, so no need to unrotate. Let me improve: use adaptive threshol ...&rdquo;</blockquote>
</details>
<details><summary><b>What Beam's program sees, including where it still fails (p06: a rotated, noisy scan)</b></summary><p>Left: the rooms (colour), doors (red dot; green = door to outside) and door edges recovered by Beam low, 10 rounds, each revises the best code so far (round 9), drawn over the plan. Right: the true room graph. The blue line is a <b>wheelchair route to the exit</b>: a networkx shortest path over the door graph from the room farthest from the exit to the outside door, computed only from the recovered map. Blue dot = start.</p>
<h3>p17 (clean)</h3><div class="figs"><div><img src="img/p17-beam.png" alt="Beam recovered graph for p17"><div>Beam</div></div><div><img src="img/p17-truth.png" alt="true graph for p17"><div>Truth</div></div></div>
<div class="cap">Rooms matched 4/4 (pred 5), doors matched 0/4 (pred 0), plan score 0.000, partial 0.444. Beam map: no route: no recovered room has a door path to outside. True route: 2 door(s), narrowest 1.04 m, all at least 0.8 m wide.</div>
<h3>p01 (outline+clutter)</h3><div class="figs"><div><img src="img/p01-beam.png" alt="Beam recovered graph for p01"><div>Beam</div></div><div><img src="img/p01-truth.png" alt="true graph for p01"><div>Truth</div></div></div>
<div class="cap">Rooms matched 3/5 (pred 10), doors matched 0/5 (pred 0), plan score 0.000, partial 0.200. Beam map: no route: no recovered room has a door path to outside. True route: 2 door(s), narrowest 0.84 m, all at least 0.8 m wide.</div>
<h3>p06 (clutter+scan)</h3><div class="figs"><div><img src="img/p06-beam.png" alt="Beam recovered graph for p06"><div>Beam</div></div><div><img src="img/p06-truth.png" alt="true graph for p06"><div>Truth</div></div></div>
<div class="cap">Rooms matched 2/5 (pred 10), doors matched 0/5 (pred 6), plan score 0.000, partial 0.133. Beam map: route found through 1 door(s); 1 of 10 recovered rooms reach the exit. True route: 3 door(s), narrowest 0.68 m, 1 under 0.8 m (too narrow for many wheelchairs).</div>
<h3>p12 (outline+scan)</h3><div class="figs"><div><img src="img/p12-beam.png" alt="Beam recovered graph for p12"><div>Beam</div></div><div><img src="img/p12-truth.png" alt="true graph for p12"><div>Truth</div></div></div>
<div class="cap">Rooms matched 2/4 (pred 2), doors matched 0/4 (pred 0), plan score 0.000, partial 0.333. Beam map: no route: no recovered room has a door path to outside. True route: 2 door(s), narrowest 0.80 m, 1 under 0.8 m (too narrow for many wheelchairs).</div>
</details>
<details><summary><b>Why this is useful</b></summary><p>Floor plans exist mostly as images. A room graph turns one into something a program can answer questions about: is every room reachable from the exit, what is the step-free route for a wheelchair, how many bedrooms, which rooms are interior. The environment is a ready-made, exactly graded RL task for teaching a text-only model to do visual work through code, with dense partial credit so early programs still get a learning signal.</p>
</details>
<details><summary><b>Credit</b></summary><p>Plan geometry: Swiss Dwellings v3.0.0, Zenodo, DOI <a href="https://doi.org/10.5281/zenodo.7788422">10.5281/zenodo.7788422</a>, CC BY 4.0. Plans re-drawn from that geometry. Built by Hamood (hamoodey) for Reflection's Build with Beam contest. Beam: <code>Beam-1.0-501B-A23B</code> via the Reflection API.</p></details>
<details><summary><b>Can the reward be gamed?</b></summary><ul>
<li>Held-out plans come from 64 buildings that appear in no practice plan; feedback in the loop uses only the 24 practice plans.</li>
<li>The program runs as an unprivileged user that owns no file, offline, with 30 s CPU per plan; it sees only <code>plan.png</code> and <code>manifest.json</code>.</li>
<li>Output is clamped: at most 2 MB, 100 rooms, 300 doors, 50,000 points; any schema error, or rooms overlapping by more than 10%, scores the plan 0.</li>
<li>Rooms are matched one to one (Hungarian) at IoU 0.7, so extra or split rooms lower F1; a door counts only if both its rooms match and its centre is within 0.5 m.</li>
<li>Spamming doors fails: a clean plan with door precision under 0.5 counts as 0 in the gate.</li>
<li>Sanity checks: the empty stub scores 0.000; the hand-written reference scores 0.242 strict.</li></ul></details>
</body></html>