update

README.md

@@ -76,6 +76,13 @@ Every prompt, response, reasoning trace, program and score per round is in `runs
 - Grade on data the feedback never saw: 64 held-out plans from other buildings keep the gain honest.
 - Score partial credit as well as strict: strict was 0.000 in every run, so only partial credit shows learning.
 
+## Beam reading its failure report (reasoning_content, Beam low, 10-round run)
+> **Round 5** (practice-plan partial credit 0.022 -> 0.179): "My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach."
+
+> **Round 7** (practice-plan partial credit 0.078 -> 0.174): "The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet."
+
+Full traces: `runs/beam-low-best10/round-5/reasoning.md`, `runs/beam-low-best10/round-7/reasoning.md`.
+
 ## Results
 | Model | Rounds | Best public strict | Best public partial | Held-out strict (64 plans) | Held-out partial | Best round |
 |---|---|---|---|---|---|---|

index.html

@@ -20,7 +20,7 @@ def floorplan_reward(completions, **kw):   # GRPO-style reward (e.g. TRL GRPOTra
 <li>Write down what worked: a 46-line tips playbook more than doubled the held-out score (0.032 &rarr; 0.075, 3 repeats each).</li>
 <li>Grade on data the feedback never saw: 64 held-out plans from other buildings keep the gain honest.</li>
 <li>Score partial credit as well as strict: strict was 0.000 in every run, so only partial credit shows learning.</li>
-</ul><h2>Why trust these numbers</h2>
+</ul><h2>Beam reading its failure report (<code>reasoning_content</code>, Beam low, 10-round run)</h2><blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em"><b>Round 5</b> (practice-plan partial credit 0.022 &rarr; 0.179): &ldquo;My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach.&rdquo;</blockquote><blockquote style="border-left:4px solid #2b8cbe;margin:.5em 0;padding:.3em 1em"><b>Round 7</b> (practice-plan partial credit 0.078 &rarr; 0.174): &ldquo;The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet.&rdquo;</blockquote><h2>Why trust these numbers</h2>
 <p>The grader is exact (rooms at IoU 0.7, doors within 0.5 m). Feedback uses only the 24 practice plans; the 64 held-out plans are from other buildings and never feed back. Every prompt, reasoning trace, program and score is in <code>runs/</code>.</p>
 <p><b>Known limits.</b> Kimi K3 did not finish (no reply to its round-1 request in 31 min); Beam high effort finished (held-out 0.016 in a single 3-round run; 0.176 with best of 4 samples per round). Most numbers are one run; the playbook result has 3 repeats with a wide spread. Doors are nearly unsolved (door F1 0.000 on every setup but one). We did not update Beam's weights.</p><details><summary><b>What this entry is</b></summary><ul><li><b>An RL environment for Beam.</b> <code>FloorplanEnv.reset()</code> gives the task prompt; <code>step(code)</code> runs Beam's <code>run.py</code> and returns the strict reward, a dense partial-credit reward and per-style feedback (room F1, door F1, top failures).</li><li><b>Practice and held-out sets.</b> 24 public practice plans give the feedback; 64 held-out plans from other buildings give the final score and never feed back.</li><li><b>Runnable.</b> Docker files for the sandbox and the grader are in the dataset (<code>task/environment</code>, <code>task/tests</code>).</li><li><b>Benchmark evidence.</b> The Beam and Kimi K3 runs below show what the environment measures.</li></ul><p>We did not update Beam's weights; in these runs Beam improves only in-context, by retrying with the environment's feedback. The environment is ready for RL fine-tuning of Beam (open weights).</p>
 </details>