update

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+img/p01-truth.png filter=lfs diff=lfs merge=lfs -text
+img/p09-beam.png filter=lfs diff=lfs merge=lfs -text
+img/p09-truth.png filter=lfs diff=lfs merge=lfs -text
+img/p20-beam.png filter=lfs diff=lfs merge=lfs -text
+img/p20-truth.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -8,8 +8,7 @@ pinned: false
 license: cc-by-4.0
 short_description: Beam learns to read real floor plans by writing code
 ---
-
-**Main finding: results are being added (Beam low, Beam high, Kimi K3 vs a reference solution at about 0.24); see the Space for live numbers.**
+**Results are being added: Beam (low and high reasoning effort) and Kimi K3 write floor-plan readers over 3 rounds of graded feedback; best so far, Beam low reaches 0.108 public partial credit (strict 0.000) vs a reference solution at about 0.24 strict.**
 
 # Floorplan Graph RL: an RL environment that teaches Beam to read real apartment floor plans by writing code
 
@@ -33,32 +32,63 @@ Beam (`Beam-1.0-501B-A23B`, Reflection API, `reasoning_effort` low and high) is
 2. Beam answers with one complete `run.py`.
 3. `env.step(code)` runs that program on the 24 public plans in a sandbox (30 s CPU per plan), scores it, and returns the reward, partial credit and a failure report (crashes, timeouts, invalid output, low IoU rooms, missed doors).
 4. Beam gets the feedback and its previous code and writes a better `run.py`. Rounds repeat.
-5. The best public round is graded once on the 64 held-out plans (buildings never seen in the public set). The held-out plans never feed back into the loop.
+5. The best public round is graded once on the 64 held-out plans (buildings never seen in the public set; published in `task/hidden/` with answers, for reproducibility). The held-out plans never feed back into the loop.
 
-Every prompt, response, reasoning trace, program and score per round is in `runs/`.
+Every prompt, response, reasoning trace, program and score per round is in `runs/`. The same loop also runs Kimi K3 as a comparison, and an earlier Opus 5.5 two-hour agent run (0.032 held-out) is listed as a baseline.
 
 ## Reward
 - **Per plan:** rooms are matched one to one (Hungarian) at IoU >= 0.7; a door counts when both its rooms match and its centre lies within 0.5 m of the true door. Plan score = room F1 x door F1.
 - **Strict reward (the official score):** 16 clean plans form a gate, g = min(1, mean clean score / 0.6); reward = g x mean score over the 48 hard plans (16 per hard style).
 - **Partial credit (dense RL signal):** mean over plans of (room F1 + door F1) / 2. The strict reward is near zero for most early programs; partial credit still moves when a program gets rooms right but misses doors, so the model sees progress.
 
+## Results
+| Model | Rounds | Best public strict | Best public partial | Held-out strict (64 plans) | Held-out partial | Best round |
+|---|---|---|---|---|---|---|
+| Beam, reasoning low | 3 | 0.000 | 0.108 | 0.000 | 0.118 | round 3 |
+| Beam, reasoning high | 1 | 0.000 | 0.000 | pending | pending | round 1 |
+| Kimi K3 | 0 | - | - | pending | pending | - |
+| Earlier baseline: Opus 5.5, one 2-hour agent run with tools (not this loop) | - | - | - | 0.032 | - | - |
+| Reference solution (hand-written, for scale) | - | - | - | about 0.24 | - | - |
+
+Strict = the official score (below). Partial = the dense RL signal. Public scores drive the feedback; the 64 held-out plans are used only once, for the final score. Per-round scores, plan overlays and the wheelchair route are on the Space.
+
 ## Layout
 | Path | What |
 |---|---|
 | `task/instruction.md` | the task statement given to the model |
 | `task/workspace/README.md` | full input/output contract |
 | `task/workspace/docs/STYLE-GUIDE.md` | drawing styles, door conventions, ranges |
-| `task/workspace/public/` | 24 public plans (6 per style): `cases/pNN/plan.png` + `manifest.json`, `answers/pNN.json` |
-| `task/workspace/selfcheck/` | self-check runner and the official scorer |
-| `task/workspace/submission/run.py` | starting stub |
-| `code/` | `env.py` (the RL environment), the model loop, grading CLIs |
+| `task/workspace/public/` | 24 public plans (6 per style): `cases/pNN/plan.png` + `manifest.json`, answers in `answers/pNN.json` |
+| `task/workspace/selfcheck/` | self-check, the exact run rules (`runner.py`) and scorer (`score.py`) |
+| `task/workspace/submission/run.py` | starting stub (scores 0) |
+| `task/tests/` | held-out grader: `verify.py`, `score.py`, the renderer `gen/`, the 64 held-out apartments (`hidden/units/`, geometry), `hidden/index.json`, the render seed and `hidden/expected.json` (hashes of every rendered plan and answer). The verifier renders the 64 held-out plans and their answers from these at grade time. |
+| `task/hidden/` | the 64 held-out plans, rendered: `hNN/plan.png`, `manifest.json`, `answer.json` (same format as the public answers), `index.json` (style, scale, counts) |
+| `task/environment/` | Docker setup of the sandbox the program runs in (Ubuntu 24.04, Python 3.12, pinned packages) |
+| `code/env.py` | `FloorplanEnv`: `reset()` returns the task prompt; `step(code)` returns (reward, partial, feedback, info) |
+| `code/grade.py` | grades a `run.py` with the Docker images (public self-check or held-out verifier); prints strict reward, partial credit, per-style F1, failures |
+| `code/llm.py`, `code/loop.py` | the model loop (Beam via the Reflection API, Kimi K3) |
+| `code/build_space.py` | builds the Space page: table, chart, overlays, networkx wheelchair route |
+| `runs/<model>/round-<k>/` | every round: `prompt.md`, `response.md`, `reasoning.md` (Beam's thinking), `run.py`, `public.json` (scores); `runs/<model>/hidden.json` final held-out score |
 
 ## How to run
 ```bash
-# self-check a program on the 24 public plans (Python 3.12, numpy, scipy, opencv, scikit-image, shapely, networkx, Pillow)
-cp my_run.py task/workspace/submission/run.py
-python3 task/workspace/selfcheck/selfcheck.py
+# 1. Build the sandbox and the held-out grader (Docker, linux/amd64)
+docker build -t fp-agent task/environment
+docker build -t fp-verifier --build-arg TASK_IMAGE=fp-agent task/tests
+
+# 2. Public self-check of your program (24 plans, official runner and scorer)
+docker run --rm -v "$PWD/my_submission:/mnt/sub:ro" --entrypoint bash fp-verifier -c \
+  'rm -rf /app/submission/* && cp -a /mnt/sub/. /app/submission/ && python3 /app/selfcheck/selfcheck.py'
+
+# 3. Held-out score (renders the 64 held-out plans, runs your program as an unprivileged user, writes /logs/verifier/reward.json)
+docker run --rm -v "$PWD/my_submission:/mnt/sub:ro" --entrypoint bash fp-verifier -c \
+  'rm -rf /app/submission/* && cp -a /mnt/sub/. /app/submission/ && /tests/test.sh; cat /logs/verifier/reward.json'
+
+# 4. The RL loop (Beam writes run.py, gets scores and failures, retries); needs REFLECTION_API_KEY
+#    code/grade.py runs the docker commands above over ssh on FP_HOST (a Docker host with the two images)
+python3 code/loop.py --model beam --effort high
 ```
+Without Docker: `cp my_run.py task/workspace/submission/run.py && python3 task/workspace/selfcheck/selfcheck.py` (needs Python 3.12 with numpy, scipy, opencv, scikit-image, shapely, networkx, Pillow).
 
 ## Credit and license
 Plan geometry: **Swiss Dwellings v3.0.0** (Standfest et al.), Zenodo, DOI [10.5281/zenodo.7788422](https://doi.org/10.5281/zenodo.7788422), CC BY 4.0.

index.html

@@ -1,7 +1,43 @@
-<!doctype html><html><head><meta charset="utf-8"><title>Floorplan Graph RL</title>
-<style>body{font-family:system-ui,sans-serif;max-width:1000px;margin:2em auto;padding:0 1em;line-height:1.5}</style></head><body>
+<!doctype html><html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Floorplan Graph RL: Beam reads real floor plans by writing code</title><style>body{font-family:system-ui,-apple-system,sans-serif;max-width:1100px;margin:2em auto;padding:0 1em;line-height:1.5;color:#222}table{border-collapse:collapse;margin:1em 0}td,th{border:1px solid #ccc;padding:6px 10px;text-align:right}td:first-child,th:first-child{text-align:left}.figs{display:grid;grid-template-columns:1fr 1fr;gap:12px}.figs img{width:100%;border:1px solid #ddd}.cap{font-size:.9em;color:#444;margin-bottom:1.5em}code{background:#f3f3f3;padding:1px 4px}.note{color:#666;font-size:.9em}</style></head><body>
 <h1>Floorplan Graph RL: Beam reads real apartment floor plans by writing code</h1>
-<p>An RL environment: Beam writes <code>run.py</code>, the environment runs it on real floor plans, scores it (strict reward + partial credit) and returns failures; Beam retries.</p>
-<p>Results, charts and plan overlays are being added. Data and code: <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl">hamoodey/floorplan-graph-rl (dataset)</a>.</p>
-<p>Plan geometry: Swiss Dwellings v3.0.0, DOI <a href="https://doi.org/10.5281/zenodo.7788422">10.5281/zenodo.7788422</a>, CC BY 4.0.</p>
-</body></html>
+<p><b>Results are being added: Beam (low and high reasoning effort) and Kimi K3 write floor-plan readers over 3 rounds of graded feedback; best so far, Beam low reaches 0.108 public partial credit (strict 0.000) vs a reference solution at about 0.24 strict.</b></p>
+<p>Beam is text-only. It cannot see a floor plan, but it can write a program that reads one. This RL environment gives Beam a raster plan of a real Swiss apartment and asks for <code>run.py</code>, a program that recovers the <b>room graph</b>: every room as a polygon, every door as an edge. The environment runs the program on 24 public plans in a sandbox, scores it, and sends back the reward, partial credit and the worst failures. Beam rewrites the program. The best round is then scored once on 64 held-out plans from other buildings.</p>
+<p>Data, environment, grader, all plans and every run (prompts, reasoning, code, scores): <a href="https://huggingface.co/datasets/hamoodey/floorplan-graph-rl">huggingface.co/datasets/hamoodey/floorplan-graph-rl</a></p>
+<h2>Results</h2><table><tr><th>Model</th><th>Rounds</th><th>Best public strict</th><th>Best public partial</th><th>Held-out strict (64 plans)</th><th>Held-out partial</th><th>Best round</th></tr>
+<tr><td>Beam, reasoning low</td><td>3</td><td>0.000</td><td>0.108</td><td>0.000</td><td>0.118</td><td>round 3</td></tr>
+<tr><td>Beam, reasoning high</td><td>1</td><td>0.000</td><td>0.000</td><td>pending</td><td>pending</td><td>round 1</td></tr>
+<tr><td>Kimi K3</td><td>0</td><td>-</td><td>-</td><td>pending</td><td>pending</td><td>-</td></tr>
+<tr><td>Earlier baseline: Opus 5.5, one 2-hour agent run with tools (not this loop)</td><td>-</td><td>-</td><td>-</td><td>0.032</td><td>-</td><td>-</td></tr>
+<tr><td>Reference solution (hand-written, for scale)</td><td>-</td><td>-</td><td>-</td><td>about 0.24</td><td>-</td><td>-</td></tr></table>
+<p class="note"><b>Strict</b> = the official score: plan score = room F1 x door F1 (rooms matched one to one at IoU &ge; 0.7, a door counts when its rooms match and its centre is within 0.5 m); gate g = min(1, mean clean / 0.6); strict = g x mean over the 48 hard plans. <b>Partial</b> = mean over plans of (room F1 + door F1) / 2, the dense RL signal. Public scores drive the feedback; held-out plans are used only for the final score.</p>
+<h2>Score per round (public plans)</h2><svg viewBox="0 0 640 300" width="100%" style="max-width:640px" xmlns="http://www.w3.org/2000/svg" font-family="system-ui" font-size="12">
+<line x1="55" x2="490" y1="260.0" y2="260.0" stroke="#ddd"/><text x="49" y="264.0" text-anchor="end">0.00</text>
+<line x1="55" x2="490" y1="212.0" y2="212.0" stroke="#ddd"/><text x="49" y="216.0" text-anchor="end">0.02</text>
+<line x1="55" x2="490" y1="164.0" y2="164.0" stroke="#ddd"/><text x="49" y="168.0" text-anchor="end">0.05</text>
+<line x1="55" x2="490" y1="116.0" y2="116.0" stroke="#ddd"/><text x="49" y="120.0" text-anchor="end">0.07</text>
+<line x1="55" x2="490" y1="68.0" y2="68.0" stroke="#ddd"/><text x="49" y="72.0" text-anchor="end">0.10</text>
+<line x1="55" x2="490" y1="20.0" y2="20.0" stroke="#ddd"/><text x="49" y="24.0" text-anchor="end">0.12</text>
+<text x="55.0" y="278" text-anchor="middle">round 1</text>
+<text x="272.5" y="278" text-anchor="middle">round 2</text>
+<text x="490.0" y="278" text-anchor="middle">round 3</text>
+<text x="14" y="130.0" transform="rotate(-90 14 130.0)" text-anchor="middle">public partial credit</text>
+<polyline fill="none" stroke="#2b8cbe" stroke-width="2.5" points="55.0,260.0 272.5,260.0 490.0,51.3"/>
+<circle cx="55.0" cy="260.0" r="4" fill="#2b8cbe"><title>Beam, reasoning low round 1: partial 0.000, strict 0.000</title></circle>
+<circle cx="272.5" cy="260.0" r="4" fill="#2b8cbe"><title>Beam, reasoning low round 2: partial 0.000, strict 0.000</title></circle>
+<circle cx="490.0" cy="51.3" r="4" fill="#2b8cbe"><title>Beam, reasoning low round 3: partial 0.108, strict 0.000</title></circle>
+<line x1="502" x2="522" y1="30" y2="30" stroke="#2b8cbe" stroke-width="3"/><text x="528" y="34">Beam, reasoning low</text>
+<polyline fill="none" stroke="#08306b" stroke-width="2.5" points="55.0,260.0"/>
+<circle cx="55.0" cy="260.0" r="4" fill="#08306b"><title>Beam, reasoning high round 1: partial 0.000, strict 0.000</title></circle>
+<line x1="502" x2="522" y1="50" y2="50" stroke="#08306b" stroke-width="3"/><text x="528" y="54">Beam, reasoning high</text>
+</svg>
+<h2>What Beam's program sees</h2><p>Left: the rooms (colour), doors (red dot; green = door to outside) and door edges recovered by Beam, reasoning low (round 3), drawn over the plan. Right: the true room graph. The blue line is a <b>wheelchair route to the exit</b>: a networkx shortest path over the door graph from the room farthest from the exit to the outside door, computed only from the recovered map. Blue dot = start.</p>
+<h3>p22 (clean)</h3><div class="figs"><div><img src="img/p22-beam.png" alt="Beam recovered graph for p22"><div>Beam</div></div><div><img src="img/p22-truth.png" alt="true graph for p22"><div>Truth</div></div></div>
+<div class="cap">Rooms matched 4/6 (pred 5), doors matched 0/6 (pred 0), plan score 0.000, partial 0.364. Beam map: no route: the recovered map has no door to outside. True route: 3 door(s), narrowest 0.67 m, 1 under 0.8 m (too narrow for many wheelchairs).</div>
+<h3>p01 (outline+clutter)</h3><div class="figs"><div><img src="img/p01-beam.png" alt="Beam recovered graph for p01"><div>Beam</div></div><div><img src="img/p01-truth.png" alt="true graph for p01"><div>Truth</div></div></div>
+<div class="cap">Rooms matched 2/5 (pred 9), doors matched 0/5 (pred 0), plan score 0.000, partial 0.143. Beam map: no route: the recovered map has no door to outside. True route: 2 door(s), narrowest 0.84 m, all at least 0.8 m wide.</div>
+<h3>p09 (clutter+scan)</h3><div class="figs"><div><img src="img/p09-beam.png" alt="Beam recovered graph for p09"><div>Beam</div></div><div><img src="img/p09-truth.png" alt="true graph for p09"><div>Truth</div></div></div>
+<div class="cap">Rooms matched 2/8 (pred 2), doors matched 0/10 (pred 0), plan score 0.000, partial 0.200. Beam map: no route: the recovered map has no door to outside. True route: 3 door(s), narrowest 0.92 m, all at least 0.8 m wide.</div>
+<h3>p20 (outline+scan)</h3><div class="figs"><div><img src="img/p20-beam.png" alt="Beam recovered graph for p20"><div>Beam</div></div><div><img src="img/p20-truth.png" alt="true graph for p20"><div>Truth</div></div></div>
+<div class="cap">Rooms matched 3/4 (pred 3), doors matched 0/4 (pred 0), plan score 0.000, partial 0.429. Beam map: no route: the recovered map has no door to outside. True route: 2 door(s), narrowest 0.84 m, all at least 0.8 m wide.</div>
+<h2>Why this is useful</h2><p>Floor plans exist mostly as images. A room graph turns one into something a program can answer questions about: is every room reachable from the exit, what is the step-free route for a wheelchair, how many bedrooms, which rooms are interior. The environment is a ready-made, exactly graded RL task for teaching a text-only model to do visual work through code, with dense partial credit so early programs still get a learning signal.</p>
+<h2>Credit</h2><p>Plan geometry: Swiss Dwellings v3.0.0, Zenodo, DOI <a href="https://doi.org/10.5281/zenodo.7788422">10.5281/zenodo.7788422</a>, CC BY 4.0. Plans re-drawn from that geometry. Built by Hamood (hamoodey) for Reflection's Build with Beam contest. Beam: <code>Beam-1.0-501B-A23B</code> via the Reflection API.</p></body></html>