README.md

metadata

title: Floorplan Graph RL
emoji: 🏠
colorFrom: blue
colorTo: green
sdk: static
pinned: false
license: cc-by-4.0
short_description: Beam learns to read real floor plans by writing code
thumbnail: >-
  https://huggingface.co/spaces/hamoodey/floorplan-graph-rl/resolve/main/img/hero.png

Beam can't see an image. So it learned to write its own eyes.

A text-only Beam agent writes its own computer-vision pipeline, learns from exact grader feedback, and improves without weight updates. Its best 10-round run reaches 0.179 held-out partial credit on 64 unseen real apartment plans and 0.803 room F1 on clean plans.

Beam actually learned: after the grader reported zero rooms on scans, Beam diagnosed its thresholding failure and adopted adaptive thresholding. Public partial credit jumped from 0.022 to 0.179.

The problem. Many floor plans exist only as images or scans. A room graph (rooms + doors) answers "can a wheelchair reach every room?", "how do I get out?" and "how many rooms?". This environment teaches a text-only model to build that graph from a real plan, and grades it exactly.

In one screen.

• A novel Beam loop: Beam writes a complete vision program, the environment executes it on real plans, and Beam uses exact rewards and failure reports to revise it. Every prompt, reasoning trace, program, and score is published.
• Measured improvement: the 10-round run rises from 0.000 to 0.182 public partial credit; the selected round scores 0.179 on 64 held-out plans from other buildings. A 46-line general tips playbook also lifts Beam low from 0.032 to 0.075 held-out (3 repeats each; wide spread).
• A reusable, unsaturated RL environment: reset() / step(code), strict and dense rewards, 24 practice + 64 held-out plans, Docker grading, and a GRPO example. Strict reward remains 0.000 versus 0.242 for the reference: rooms are emerging; doors are the open research problem.

Evidence	Result
Beam low, 10 feedback rounds	0.000 → 0.182 public partial; 0.179 held-out partial
General tips playbook, 3 repeats each	0.032 → 0.075 held-out partial
Clean-plan room recovery, best 10-round run	0.803 room F1
Strict held-out reward	0.000 for every Beam setup; 0.242 reference

Floorplan Graph RL: an RL environment that teaches Beam to read real apartment floor plans by writing code

Space (results, charts, plan overlays, wheelchair route): https://huggingface.co/spaces/hamoodey/floorplan-graph-rl Dataset (this repo): https://huggingface.co/datasets/hamoodey/floorplan-graph-rl

What this entry is

• An RL environment for Beam. FloorplanEnv.reset() gives the task prompt; step(code) runs Beam's run.py and returns the strict reward, a dense partial-credit reward and per-style feedback (room F1, door F1, top failures).
• Practice and held-out sets. 24 public practice plans give the feedback; 64 held-out plans from other buildings give the final score and never feed back.
• Runnable. Docker files for the sandbox and the grader are included (see How to run).
• Benchmark evidence. The Beam and Kimi K3 runs below show what the environment measures.

We did not update Beam's weights; in these runs Beam improves only in-context, by retrying with the environment's feedback. The environment is ready for RL fine-tuning of Beam (open weights).

What it is

A text-only model cannot see an image. It can still read one, by writing a program that does. This environment gives the model a raster floor plan of a real apartment and asks for a Python program, run.py, that recovers the room graph: every room as a pixel polygon and every door as an edge between two rooms (or a room and outside). Plans come from real Swiss apartments, drawn in four styles: clean; mixed wall drawings with furniture, labels and dimension chains; clutter on a rotated, blurred, noisy scan; and mixed walls on such a scan.

A recovered room graph is useful on its own: it answers "how do I get from the bedroom to the exit", "is every room reachable through doors wide enough for a wheelchair", "how many rooms does this flat have". The Space contrasts routes from Beam's recovered graph with the true graph; Beam's door recovery is not yet reliable enough for a trustworthy route.

How Beam is used

Beam (Beam-1.0-501B-A23B, Reflection API, reasoning_effort low and high) is the policy:

1. env.reset() returns the task prompt: the instruction, the full I/O contract (task/workspace/README.md) and the style guide (task/workspace/docs/STYLE-GUIDE.md).
2. Beam answers with one complete run.py.
3. env.step(code) runs that program on the 24 public plans in a sandbox (30 s CPU per plan), scores it, and returns the reward, partial credit and a failure report (crashes, timeouts, invalid output, low IoU rooms, missed doors).
4. Beam gets the feedback and its previous code and writes a better run.py. Rounds repeat.
5. The best public round is graded once on the 64 held-out plans (buildings never seen in the public set; published in task/hidden/ with answers, for reproducibility). The held-out plans never feed back into the loop.

Every prompt, response, reasoning trace, program and score per round is in runs/. The same loop also runs Kimi K3 as a comparison, and an earlier Opus 5.5 two-hour agent run (0.032 held-out) is listed as a baseline.

Reward

• Per plan: rooms are matched one to one (Hungarian) at IoU >= 0.7; a door counts when both its rooms match and its centre lies within 0.5 m of the true door. Plan score = room F1 x door F1.
• Strict reward (the official score): 16 clean plans form a gate, g = min(1, mean clean score / 0.6); reward = g x mean score over the 48 hard plans (16 per hard style).
• Partial credit (dense RL signal): mean over plans of (room F1 + door F1) / 2. The strict reward is near zero for most early programs; partial credit still moves when a program gets rooms right but misses doors, so the model sees progress.

What generalizes for anyone using Beam on code-writing tasks

• Feedback beats a single try: graded retries took Beam from 0.000 (round 1) to 0.179 held-out partial credit with no weight change.
• Write down what worked: a 46-line tips playbook more than doubled the held-out score (0.032 -> 0.075, 3 repeats each).
• Grade on data the feedback never saw: 64 held-out plans from other buildings keep the gain honest.
• Score partial credit as well as strict: strict was 0.000 in every run, so only partial credit shows learning.

Beam reading its failure report (reasoning_content, Beam low, 10-round run)

Round 5 (practice-plan partial credit 0.022 -> 0.179): "My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach."

Round 7 (practice-plan partial credit 0.078 -> 0.174): "The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet."

Full traces: runs/beam-low-best10/round-5/reasoning.md, runs/beam-low-best10/round-7/reasoning.md.

Results

Model	Rounds	Best public strict	Best public partial	Held-out strict (64 plans)	Held-out partial	Best round
Beam low effort, 3 rounds (repeat 1)	3	0.000	0.000	0.000	0.000	round 1
Beam low effort, 3 rounds (repeat 2)	3	0.000	0.026	0.000	0.035	round 3
Beam low effort, 3 rounds (repeat 3)	3	0.000	0.058	0.000	0.061	round 2
Beam low + playbook, 3 rounds (repeat 1)	3	0.000	0.101	0.000	0.102	round 1
Beam low + playbook, 3 rounds (repeat 2)	3	0.000	0.000	0.000	0.000	round 1
Beam low + playbook, 3 rounds (repeat 3)	3	0.000	0.136	0.000	0.123	round 1
Beam high effort, 3 rounds	3	0.000	0.010	0.000	0.016	round 2
Beam high + playbook, 3 rounds	3	0.000	0.000	0.000	0.000	round 1
Beam low, 10 rounds, each revises the best code so far	10	0.000	0.182	0.000	0.179	round 9
Beam high, 5 rounds, best of 4 samples per round	5	0.000	0.181	0.000	0.176	round 4
Earlier baseline: Opus 5.5, one 2-hour agent run with tools (a different setup; strict score)	-	-	-	0.032	-	-
Reference solution (hand-written, for scale)	-	-	-	0.242	0.586	-

Strict = the official score (below). Partial = the dense RL signal. Public scores drive the feedback; the 64 held-out plans are used only once, for the final score. Per-round scores, plan overlays and the wheelchair route are on the Space.

Layout

Path	What
task/instruction.md	the task statement given to the model
task/workspace/README.md	full input/output contract
task/workspace/docs/STYLE-GUIDE.md	drawing styles, door conventions, ranges
task/workspace/public/	24 public plans (6 per style): cases/pNN/plan.png + manifest.json, answers in answers/pNN.json
task/workspace/selfcheck/	self-check, the exact run rules (runner.py) and scorer (score.py)
task/workspace/submission/run.py	starting stub (scores 0)
task/tests/	held-out grader: verify.py, score.py, the renderer gen/, the 64 held-out apartments (hidden/units/, geometry), hidden/index.json, the render seed and hidden/expected.json (hashes of every rendered plan and answer). The verifier renders the 64 held-out plans and their answers from these at grade time.
task/hidden/	the 64 held-out plans, rendered: hNN/plan.png, manifest.json, answer.json (same format as the public answers), index.json (style, scale, counts)
task/environment/	Docker setup of the sandbox the program runs in (Ubuntu 24.04, Python 3.12, pinned packages)
code/env.py	FloorplanEnv: reset() returns the task prompt; step(code) returns (reward, partial, feedback, info)
code/grade.py	grades a run.py with the Docker images (public self-check or held-out verifier); prints strict reward, partial credit, per-style F1, failures
code/llm.py, code/loop.py	the model loop (Beam via the Reflection API, Kimi K3)
code/build_space.py	builds the Space page: table, chart, overlays, networkx wheelchair route
runs/<model>/round-<k>/	every round: prompt.md, response.md, reasoning.md (Beam's thinking), run.py, public.json (scores); runs/<model>/hidden.json final held-out score

Use it for RL

from code.env import FloorplanEnv, extract_code   # run from the repo root; grading needs the two Docker images (How to run)
env = FloorplanEnv(set="public")                  # 24 practice plans; set="hidden" only for the final score
prompt = env.reset()                              # instruction + contract + style guide + answer format
reward, partial, feedback, info = env.step(extract_code(model_reply))   # strict reward, dense reward, per-style report
next_prompt = prompt + "\n\nYour previous run.py:\n" + code + "\n\nGrader feedback:\n" + feedback

A GRPO-style reward function (for example TRL's GRPOTrainer(reward_funcs=[floorplan_reward])), using the dense partial credit so that early programs still get a signal:

def floorplan_reward(completions, **kwargs):
    out = []
    for c in completions:
        text = c if isinstance(c, str) else c[-1]["content"]
        reward, partial, feedback, info = FloorplanEnv(set="public").step(extract_code(text))
        out.append(partial + reward)   # strict reward adds on top once door F1 > 0
    return out

The task is far from saturated: the strict reward is still 0.000 for every model loop here, while a hand-written reference reaches 0.242. There is a lot of room to train.

Why trust these numbers

The grader is exact (rooms at IoU 0.7, doors within 0.5 m). Feedback uses only the 24 practice plans; the 64 held-out plans are from other buildings and never feed back. Every prompt, reasoning trace, program and score is in runs/.

Known limits. Kimi K3 did not finish (no reply to its round-1 request in 31 min); Beam high effort finished (held-out 0.016 in a single 3-round run; 0.176 with best of 4 samples per round). Most numbers are one run; the playbook result has 3 repeats with a wide spread. Doors are nearly unsolved (door F1 0.000 on every setup but one). We did not update Beam's weights.

How to run

# 1. Build the sandbox and the held-out grader (Docker, linux/amd64)
docker build -t fp-agent task/environment
docker build -t fp-verifier --build-arg TASK_IMAGE=fp-agent task/tests

# 2. Public self-check of your program (24 plans, official runner and scorer)
docker run --rm -v "$PWD/my_submission:/mnt/sub:ro" --entrypoint bash fp-verifier -c \
  'rm -rf /app/submission/* && cp -a /mnt/sub/. /app/submission/ && python3 /app/selfcheck/selfcheck.py'

# 3. Held-out score (renders the 64 held-out plans, runs your program as an unprivileged user, writes /logs/verifier/reward.json)
docker run --rm -v "$PWD/my_submission:/mnt/sub:ro" --entrypoint bash fp-verifier -c \
  'rm -rf /app/submission/* && cp -a /mnt/sub/. /app/submission/ && /tests/test.sh; cat /logs/verifier/reward.json'

# 4. The RL loop (Beam writes run.py, gets scores and failures, retries); needs REFLECTION_API_KEY
#    code/grade.py runs the docker commands above over ssh on FP_HOST (a Docker host with the two images)
python3 code/loop.py --model beam --effort high

Without Docker: cp my_run.py task/workspace/submission/run.py && python3 task/workspace/selfcheck/selfcheck.py (needs Python 3.12 with numpy, scipy, opencv, scikit-image, shapely, networkx, Pillow).

Credit and license

Plan geometry: Swiss Dwellings v3.0.0 (Standfest et al.), Zenodo, DOI 10.5281/zenodo.7788422, CC BY 4.0. The plans here are re-rendered from that geometry. This repo is released under CC BY 4.0. Built by Hamood (hamoodey) for Reflection's "Build with Beam" contest.