update

index.html

@@ -1,6 +1,6 @@
 <!doctype html><html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Floorplan Graph RL: Beam reads real floor plans by writing code</title><style>body{font-family:system-ui,-apple-system,sans-serif;max-width:1100px;margin:2em auto;padding:0 1em;line-height:1.5;color:#222}table{border-collapse:collapse;margin:1em 0}td,th{border:1px solid #ccc;padding:6px 10px;text-align:right}td:first-child,th:first-child{text-align:left}.figs{display:grid;grid-template-columns:1fr 1fr;gap:12px}.figs img{width:100%;border:1px solid #ddd}.cap{font-size:.9em;color:#444;margin-bottom:1.5em}code{background:#f3f3f3;padding:1px 4px}.note{color:#666;font-size:.9em}details{margin:.6em 0;border:1px solid #e3e3e3;border-radius:6px;padding:.4em .8em}summary{cursor:pointer;font-size:1.1em}</style></head><body>
 <h1>Floorplan Graph RL: Beam reads real apartment floor plans by writing code</h1>
-<p style='font-size:1.15em'><b>Graded feedback lifts Beam (low effort) from 0.000 partial credit in round 1 to 0.182 on the practice plans and 0.179 on 64 held-out plans.</b></p><p><img src="img/hero.png" style="width:100%" alt="input plan, true room graph, Beam graph, route to exit"></p><p class="note">Plan p02 (outline+scan), a held-out-style hard drawing. Panel 3: rooms recovered by Beam's own program (Beam low, round 9). Panel 4: the step-free route to the exit, a networkx shortest path over the true door graph; this is what a recovered graph is for. Beam's doors are not right yet (door F1 0), so its own route would not be trustworthy.</p><p>An RL environment: Beam (text-only) writes <code>run.py</code> to read real apartment floor plans; the environment runs it, scores rooms and doors (strict reward + dense partial credit), and returns per-style failures.<br>Graded feedback lifts Beam (low effort) from 0.000 partial credit in round 1 to 0.182 on the practice plans and 0.179 on 64 held-out plans after 10 rounds; room F1 0.80 on clean held-out plans; doors are the open problem. Unsaturated RL environment: strict reward is still 0.000 (hand-written reference 0.242), so there is room to train. Kimi K3 did not finish (its first request got no reply in 31 min).<br>Use it for RL: <code>prompt = env.reset(); reward, partial, feedback, info = env.step(code)</code> (snippet below). Data, grader, Docker and every run: <a href='https://huggingface.co/datasets/hamoodey/floorplan-graph-rl'>dataset</a>.</p><div style="display:flex;gap:16px;flex-wrap:wrap;align-items:flex-start"><img src="img/rounds.gif" style="max-width:460px;width:100%;border:1px solid #ddd" alt="Beam rooms per round"><div style="flex:1;min-width:300px"><img src="img/bars.png" style="width:100%" alt="held-out partial credit by setup"></div></div><p><img src="img/rounds-chart.png" style="width:100%" alt="score per round"></p><h2>Use it for RL</h2><pre style="background:#f6f6f6;padding:1em;overflow-x:auto"><code>from code.env import FloorplanEnv, extract_code
+<p style='font-size:1.15em'><b>Graded feedback lifts Beam from 0.000 to 0.179 held-out partial credit in 10 rounds (strict still 0.000; hand-written reference 0.242): an unsaturated RL environment with room to train.</b></p><p><img src="img/hero.png" style="width:100%" alt="input plan, true room graph, Beam graph, route to exit"></p><p class="note">Plan p17 (clean style). Panel 3: rooms recovered by Beam's own program (Beam low, round 9). Panel 4: the step-free route to the exit, a networkx shortest path over the true door graph; this is what a recovered graph is for. Beam's doors are not right yet (door F1 0), so its own route would not be trustworthy.</p><p>An RL environment: Beam (text-only) writes <code>run.py</code> to read real apartment floor plans; the environment runs it, scores rooms and doors (strict reward + dense partial credit), and returns per-style failures.<br>Room F1 reaches 0.80 on clean held-out plans; doors are the open problem.<br>Use it for RL: <code>prompt = env.reset(); reward, partial, feedback, info = env.step(code)</code> (snippet below). Data, grader, Docker and every run: <a href='https://huggingface.co/datasets/hamoodey/floorplan-graph-rl'>dataset</a>.</p><div style="display:flex;gap:16px;flex-wrap:wrap;align-items:flex-start"><img src="img/rounds.gif" style="max-width:460px;width:100%;border:1px solid #ddd" alt="Beam rooms per round"><div style="flex:1;min-width:300px"><img src="img/bars.png" style="width:100%" alt="held-out partial credit by setup"></div></div><p><img src="img/rounds-chart.png" style="width:100%" alt="score per round"></p><h2>Use it for RL</h2><pre style="background:#f6f6f6;padding:1em;overflow-x:auto"><code>from code.env import FloorplanEnv, extract_code
 env = FloorplanEnv(set="public")        # 24 practice plans; "hidden" only for the final score
 prompt = env.reset()                    # instruction + contract + style guide
 reward, partial, feedback, info = env.step(extract_code(model_reply))
@@ -22,6 +22,7 @@ def floorplan_reward(completions, **kw):   # GRPO-style reward (e.g. TRL GRPOTra
 <tr><td>Beam high + playbook, 3 rounds</td><td>3</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>round 1</td></tr>
 <tr><td>Beam low, 10 rounds, each revises the best code so far</td><td>10</td><td>0.000</td><td>0.182</td><td>0.000</td><td>0.179</td><td>round 9</td></tr>
 <tr><td>Beam high, 5 rounds, best of 4 samples per round</td><td>5</td><td>0.000</td><td>0.181</td><td>0.000</td><td>0.176</td><td>round 4</td></tr>
+<tr><td colspan="7" style="text-align:left">Kimi K3: no reply to round 1 in 31 min; did not finish.</td></tr>
 <tr><td>Earlier baseline: Opus 5.5, one 2-hour agent run with tools (a different setup; strict score)</td><td>-</td><td>-</td><td>-</td><td>0.032</td><td>-</td><td>-</td></tr>
 <tr><td>Reference solution (hand-written, for scale)</td><td>-</td><td>-</td><td>-</td><td>0.242</td><td>0.586</td><td>-</td></tr></table>
 <p class="note"><b>Strict</b> = the official score: plan score = room F1 x door F1 (rooms matched one to one at IoU &ge; 0.7, a door counts when its rooms match and its centre is within 0.5 m); gate g = min(1, mean clean / 0.6); strict = g x mean over the 48 hard plans. <b>Partial</b> = mean over plans of (room F1 + door F1) / 2, the dense RL signal. Public scores drive the feedback; held-out plans are used only for the final score.</p>
@@ -40,9 +41,9 @@ def floorplan_reward(completions, **kw):   # GRPO-style reward (e.g. TRL GRPOTra
 <tr><td>Reference solution</td><td>0.969</td><td>0.524</td><td>0.779</td><td>0.651</td><td>0.441</td><td>0.242</td><td>0.586</td></tr>
 </table>
 </details>
-<details><summary><b>Does a playbook in the prompt help? (held-out, mean &plusmn; std over repeats)</b></summary><p>The "playbook" is a 46-line list of general computer-vision techniques added to the prompt (<code>code/playbook.md</code>); no answers, no plan-specific hints.</p><table><tr><th>Group</th><th>Runs</th><th>Held-out strict</th><th>Held-out partial</th></tr>
-<tr><td>beam-low (no playbook)</td><td>3</td><td>0.000 &plusmn; 0.000</td><td>0.032 &plusmn; 0.025</td></tr>
-<tr><td>beam-low + playbook</td><td>3</td><td>0.000 &plusmn; 0.000</td><td>0.075 &plusmn; 0.054</td></tr>
+<details><summary><b>Does a playbook in the prompt help? (held-out, 3 repeats each)</b></summary><p>The "playbook" is a 46-line list of general computer-vision techniques added to the prompt (<code>code/playbook.md</code>); no answers, no plan-specific hints. With tips the mean is about 2.4x higher, but n = 3 and one repeat in each group scored 0, so this is not a firm result.</p><table><tr><th>Group</th><th>Runs</th><th>Held-out strict</th><th>Held-out partial</th></tr>
+<tr><td>beam-low (no playbook)</td><td>3</td><td>mean 0.000, spread 0.000 to 0.000 (runs: 0.000, 0.000, 0.000)</td><td>mean 0.032, spread 0.000 to 0.061 (runs: 0.000, 0.035, 0.061)</td></tr>
+<tr><td>beam-low + playbook</td><td>3</td><td>mean 0.000, spread 0.000 to 0.000 (runs: 0.000, 0.000, 0.000)</td><td>mean 0.075, spread 0.000 to 0.123 (runs: 0.102, 0.000, 0.123)</td></tr>
 </table>
 </details>
 <details><summary><b>Score per round (public plans)</b></summary><svg viewBox="0 0 900 340" width="100%" style="max-width:900px" xmlns="http://www.w3.org/2000/svg" font-family="system-ui" font-size="12">
@@ -122,7 +123,7 @@ def floorplan_reward(completions, **kw):   # GRPO-style reward (e.g. TRL GRPOTra
 <blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em;color:#333"><b>Round 5</b> (public partial 0.022 &rarr; 0.179): &ldquo;My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach. Let me think about what&#x27;s needed: detect walls, close small gaps, find free regions. The previous code closed gaps with morphology but maybe threshold was wrong, or morphology too aggressive. Given the complexity and time, I&#x27;ll improve: use Otsu threshold, invert properly, handle scan styles with adaptive thresholding. A ...&rdquo;</blockquote>
 <blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em;color:#333"><b>Round 7</b> (public partial 0.078 &rarr; 0.174): &ldquo;The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet. Key: rooms IoU low. Need better thresholding, keep wall gaps. Let me write improved version using Otsu + closing small gaps under 0.5m only, proper free space. Also handle scan rotation? The polygons move with image, so no need to unrotate. Let me improve: use adaptive threshol ...&rdquo;</blockquote>
 </details>
-<details><summary><b>What Beam's program sees</b></summary><p>Left: the rooms (colour), doors (red dot; green = door to outside) and door edges recovered by Beam low, 10 rounds, each revises the best code so far (round 9), drawn over the plan. Right: the true room graph. The blue line is a <b>wheelchair route to the exit</b>: a networkx shortest path over the door graph from the room farthest from the exit to the outside door, computed only from the recovered map. Blue dot = start.</p>
+<details><summary><b>What Beam's program sees, including where it still fails (p06: a rotated, noisy scan)</b></summary><p>Left: the rooms (colour), doors (red dot; green = door to outside) and door edges recovered by Beam low, 10 rounds, each revises the best code so far (round 9), drawn over the plan. Right: the true room graph. The blue line is a <b>wheelchair route to the exit</b>: a networkx shortest path over the door graph from the room farthest from the exit to the outside door, computed only from the recovered map. Blue dot = start.</p>
 <h3>p17 (clean)</h3><div class="figs"><div><img src="img/p17-beam.png" alt="Beam recovered graph for p17"><div>Beam</div></div><div><img src="img/p17-truth.png" alt="true graph for p17"><div>Truth</div></div></div>
 <div class="cap">Rooms matched 4/4 (pred 5), doors matched 0/4 (pred 0), plan score 0.000, partial 0.444. Beam map: no route: no recovered room has a door path to outside. True route: 2 door(s), narrowest 1.04 m, all at least 0.8 m wide.</div>
 <h3>p01 (outline+clutter)</h3><div class="figs"><div><img src="img/p01-beam.png" alt="Beam recovered graph for p01"><div>Beam</div></div><div><img src="img/p01-truth.png" alt="true graph for p01"><div>Truth</div></div></div>