preflight-experiments / index.html
chaleong's picture
Preflight experiments
5946ddd verified
Raw History Blame Contribute Delete
38.3 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<title>Preflight Generalisation Ledger</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Archivo+Narrow:wght@500;700&family=Public+Sans:wght@400;600&family=JetBrains+Mono:wght@400;600&display=swap">
<style>
/* Layout: a lab ledger. Thesis and scoreboard first, then one chart per question, then every verdict. */
:root {
--bg: #090a0f; --surface: #12141c; --fg: #f3f4f6; --muted: #9aa1b1; --line: #2a2e3d;
--accent: #00ffaa; --good: #00ffaa; --bad: #ff6b5e; --warn: #f5a524; --base: #6b7385; --on: #05110b;
--c0: #ff6b5e; --c1: #f5a524; --c2: #e6cf62; --c3: #6cc0d8; --c4: #a497ec; --c5: #6b7385;
--display: "Archivo Narrow", "Arial Narrow", sans-serif;
--body: "Public Sans", sans-serif;
--mono: "JetBrains Mono", ui-monospace, monospace;
color-scheme: dark;
}
body { background: var(--bg); color: var(--fg); font: 15px/1.6 var(--body); padding-inline: 16px; }
main { max-width: 1320px; margin: 0 auto; padding-block: 48px 72px; display: grid; gap: 56px; }
h1, h2 { font-family: var(--display); text-wrap: balance; margin: 0; letter-spacing: 0.01em; }
h1 { font-size: clamp(2.4rem, 6vw, 4rem); line-height: .98; }
h2 { font-size: clamp(1.6rem, 3vw, 2.2rem); line-height: 1; margin-bottom: 14px; }
p { margin: 0 0 12px; max-width: 74ch; }
.eyebrow { font: 600 11px/1 var(--mono); text-transform: uppercase; letter-spacing: 0.14em; color: var(--accent); margin-bottom: 12px; }
.lede { font-size: 1.08rem; color: var(--muted); }
.takeaway { font-weight: 600; }
.headline { display: flex; flex-wrap: wrap; gap: 12px 36px; margin-top: 20px; font-family: var(--mono); }
.headline div { display: grid; gap: 2px; }
.headline b { font-size: 1.6rem; font-variant-numeric: tabular-nums; }
.headline span { font-size: 12px; color: var(--muted); }
.scroll { overflow-x: auto; }
.chart { background: var(--surface); border: 1px solid var(--line); border-radius: 4px; padding: 24px; }
.mermaid { background: var(--surface); border: 1px solid var(--line); border-radius: 4px; padding: 24px; overflow-x: auto; }
main > * { min-width: 0; }
svg text { fill: var(--muted); font: 11px var(--mono); }
svg .lab { fill: var(--fg); } svg .strong { font-weight: 600; } svg .on { fill: var(--on); font-weight: 600; }
svg .grid { stroke: var(--line); stroke-dasharray: 2 4; } svg .axis { stroke: var(--line); }
svg .ref { stroke: var(--fg); stroke-dasharray: 4 3; opacity: 0.6; }
.b-good { fill: var(--good); } .b-bad { fill: var(--bad); } .b-base { fill: var(--base); }
.d-sig { stroke: var(--good); } .d-up { stroke: var(--accent); } .d-down { stroke: var(--bad); }
.d-sig-dot { fill: var(--good); } .d-up-dot { fill: var(--accent); } .d-down-dot { fill: var(--bad); } .dot-base { fill: var(--base); }
.f0 { fill: var(--c0); } .f1 { fill: var(--c1); } .f2 { fill: var(--c2); } .f3 { fill: var(--c3); } .f4 { fill: var(--c4); } .f5 { fill: var(--c5); }
.fn0 { fill: var(--base); } .fn1 { fill: var(--good); } .fn2 { fill: var(--c1); } .fn3 { fill: var(--c3); } .fn4 { fill: var(--accent); }
.l0 { stroke: var(--accent); } .l1 { stroke: var(--c1); } .l2 { stroke: var(--base); }
.l0-dot { fill: var(--accent); } .l1-dot { fill: var(--c1); } .l2-dot { fill: var(--base); }
.ledger { display: grid; border-top: 1px solid var(--line); }
.row { display: grid; grid-template-columns: 7.5rem 1fr; gap: 4px 20px; padding-block: 14px; border-bottom: 1px solid var(--line); }
.row > div { min-width: 0; }
.row strong { display: block; }
.pill { font: 600 11px/1 var(--mono); text-transform: uppercase; letter-spacing: 0.08em; padding: 5px 8px; border-radius: 3px; border: 1px solid currentColor; justify-self: start; align-self: start; }
.promoted { color: var(--good); } .rejected { color: var(--bad); } .running { color: var(--warn); } .flat { color: var(--muted); }
@media (max-width: 560px) { .row { grid-template-columns: 1fr; } }
ul.plain { margin: 0; padding-left: 1.1rem; max-width: 68ch; } ul.plain li { margin-bottom: 6px; }
a { color: var(--accent); } a:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
footer { font-size: 13px; color: var(--muted); }
.links { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 18px; }
.links a { font: 600 12px/1 var(--mono); padding: 7px 10px; border: 1px solid var(--line); border-radius: 999px; text-decoration: none; background: var(--surface); }
</style>
<main>
<header>
<div class="eyebrow">Preflight Β· SO-101 Β· MuJoCo twin Β· updated 18 Oct 2026, 15:30 AEDT</div>
<h1>Fifteen attempts to make a GR00T arm policy generalise. Two passed the gate; a new task can be taught, but not for free.</h1>
<p class="lede">Every candidate runs the same six suites (12 scenes Γ— 8 perturbed rehearsals, seed 42) and is compared with v5 scene by scene (paired McNemar). It is promoted only with a pooled gain and no suite significantly worse, a gate fixed before the first round was scored. Everything is simulation; the physical arm has not arrived.</p>
<div class="headline">
<div><b>483 / 576</b><span>DAgger round 3, promoted (served)</span></div>
<div><b>438 / 576</b><span>v5 baseline</span></div>
<div><b>61 / 96</b><span>held-out objects, DAgger r3 (was 47–54)</span></div>
<div><b>6 β†’ 86 / 96</b><span>new task (basket) after 400 demos; the held-out ring stayed 0</span></div>
<div><b>0–1 / 50</b><span>so101-benchmark, every policy</span></div>
</div>
<nav class="links">
<a href="https://huggingface.co/spaces/chaleong/preflight" target="_blank" rel="noopener">Preflight Space: the live gate</a>
<a href="https://huggingface.co/spaces/chaleong/autopilot" target="_blank" rel="noopener">Autopilot Space: the training loop</a>
<a href="https://huggingface.co/collections/chaleong/preflight-rehearse-before-the-robot-moves-6ad409a5fccdebd3c38eea7e" target="_blank" rel="noopener">Hugging Face collection</a>
<a href="https://wandb.ai/chaleong/preflight/reports/Preflight:-SO-101-generalisation-experiments--VmlldzoxODA5MDQ3Mg==" target="_blank" rel="noopener">W&amp;B report</a>
<a href="https://github.com/charleneleong-ai/nebius-physical-ai" target="_blank" rel="noopener">Code</a>
</nav>
</header>
<section>
<h2>Scoreboard</h2>
<p class="takeaway">DAgger rounds 2 and 3 each cleared the gate: 464, then 483 on the retrain (the first round-3 run scored 481). Rounds 4 and 5, with RL-expert recoveries at Γ—8 and Γ—2, dropped back to 471 and 455. Real co-training (v9) loses 65; FLUX, trained on the same demos for the same volume, collapses. Since then, calibrated co-training (v11) is neutral at 472, failure-state synthesis falls to 437, and Autopilot's first cycle (441) traded stacking for a new task. Colours compare with v5; only the gate decides promotion.</p>
<div class="chart"><div class="scroll"><svg viewBox="0 0 640 430" width="100%" style="min-width:520px" role="img" aria-label="Pooled successes of 576 per candidate"><text x="188" y="24" text-anchor="end" class="lab">v5</text><rect x="196" y="10" width="315.4" height="20" class="b-base"/><text x="517.4" y="24" class="lab">438</text><text x="188" y="54" text-anchor="end" class="lab">v6 targeted demos</text><rect x="196" y="40" width="303.1" height="20" class="b-base"/><text x="505.1" y="54" class="lab">421</text><text x="188" y="84" text-anchor="end" class="lab">DAgger r1</text><rect x="196" y="70" width="313.9" height="20" class="b-base"/><text x="515.9" y="84" class="lab">436</text><text x="188" y="114" text-anchor="end" class="lab">DAgger r2</text><rect x="196" y="100" width="334.1" height="20" class="b-good"/><text x="536.1" y="114" class="lab">464</text><text x="188" y="144" text-anchor="end" class="lab">DAgger r3</text><rect x="196" y="130" width="346.3" height="20" class="b-good"/><text x="548.3" y="144" class="lab">481</text><text x="188" y="174" text-anchor="end" class="lab">v9 real co-training</text><rect x="196" y="160" width="268.6" height="20" class="b-bad"/><text x="470.6" y="174" class="lab">373</text><text x="188" y="204" text-anchor="end" class="lab">DAgger r3 retrain</text><rect x="196" y="190" width="347.8" height="20" class="b-good"/><text x="549.8" y="204" class="lab">483</text><text x="188" y="234" text-anchor="end" class="lab">DAgger r4 (RL recoveries)</text><rect x="196" y="220" width="339.1" height="20" class="b-good"/><text x="541.1" y="234" class="lab">471</text><text x="188" y="264" text-anchor="end" class="lab">DAgger r5 (RL x2)</text><rect x="196" y="250" width="327.6" height="20" class="b-good"/><text x="529.6" y="264" class="lab">455</text><text x="188" y="294" text-anchor="end" class="lab">FLUX LoRA (fair)</text><rect x="196" y="280" width="36.0" height="20" class="b-bad"/><text x="238.0" y="294" class="lab">50</text><text x="188" y="324" text-anchor="end" class="lab">v11 calibrated cotrain</text><rect x="196" y="310" width="339.8" height="20" class="b-good"/><text x="541.8" y="324" class="lab">472</text><text x="188" y="354" text-anchor="end" class="lab">Failure synthesis (r6)</text><rect x="196" y="340" width="314.6" height="20" class="b-base"/><text x="516.6" y="354" class="lab">437</text><text x="188" y="384" text-anchor="end" class="lab">Autopilot 1: multi-task</text><rect x="196" y="370" width="317.5" height="20" class="b-good"/><text x="519.5" y="384" class="lab">441</text><line x1="511.36" y1="4" x2="511.36" y2="404" class="ref"/><text x="515.36" y="416">v5 = 438</text><text x="610.72" y="416" text-anchor="end">of 576</text></svg></div></div>
</section>
<section>
<h2>Where round 2 gained</h2>
<p class="takeaway">Better on five of six suites, significant on one; held-out objects did not move.</p>
<p>Grey dot is v5, coloured dot is round 2. The labels count scenes where only one checkpoint succeeded.</p>
<div class="chart"><div class="scroll"><svg viewBox="0 0 720 254" width="100%" style="min-width:520px" role="img" aria-label="Per-suite successes, v5 to DAgger round 2"><text x="200.0" y="12" text-anchor="middle">40</text><line x1="200.0" y1="18" x2="200.0" y2="228" class="grid"/><text x="330.0" y="12" text-anchor="middle">60</text><line x1="330.0" y1="18" x2="330.0" y2="228" class="grid"/><text x="460.0" y="12" text-anchor="middle">80</text><line x1="460.0" y1="18" x2="460.0" y2="228" class="grid"/><text x="564.0" y="12" text-anchor="middle">96</text><line x1="564.0" y1="18" x2="564.0" y2="228" class="grid"/><text x="190" y="44" text-anchor="end" class="lab">held_out_objects</text><line x1="278.0" y1="40" x2="265.0" y2="40" class="d-down" stroke-width="3"/><circle cx="278.0" cy="40" r="5" class="dot-base"/><circle cx="265.0" cy="40" r="5" class="d-down-dot"/><text x="292.0" y="44">+12/βˆ’14 p=0.85</text><text x="190" y="78" text-anchor="end" class="lab">seen_vocabulary</text><line x1="427.5" y1="74" x2="434.0" y2="74" class="d-up" stroke-width="3"/><circle cx="427.5" cy="74" r="5" class="dot-base"/><circle cx="434.0" cy="74" r="5" class="d-up-dot"/><text x="448.0" y="78">+8/βˆ’7 p=1.0</text><text x="190" y="112" text-anchor="end" class="lab">clean_many</text><line x1="466.5" y1="108" x2="499.0" y2="108" class="d-up" stroke-width="3"/><circle cx="466.5" cy="108" r="5" class="dot-base"/><circle cx="499.0" cy="108" r="5" class="d-up-dot"/><text x="513.0" y="112">+7/βˆ’2 p=0.18</text><text x="190" y="146" text-anchor="end" class="lab">clean_top_down_many</text><line x1="479.5" y1="142" x2="492.5" y2="142" class="d-up" stroke-width="3"/><circle cx="479.5" cy="142" r="5" class="dot-base"/><circle cx="492.5" cy="142" r="5" class="d-up-dot"/><text x="506.5" y="146">+6/βˆ’4 p=0.75</text><text x="190" y="180" text-anchor="end" class="lab">real_1_pomade_tin_many</text><line x1="421.0" y1="176" x2="499.0" y2="176" class="d-sig" stroke-width="3"/><circle cx="421.0" cy="176" r="5" class="dot-base"/><circle cx="499.0" cy="176" r="5" class="d-sig-dot"/><text x="513.0" y="180">+14/βˆ’2 p=0.004</text><text x="190" y="214" text-anchor="end" class="lab">in_path_pomade_tin_many</text><line x1="414.5" y1="210" x2="466.5" y2="210" class="d-up" stroke-width="3"/><circle cx="414.5" cy="210" r="5" class="dot-base"/><circle cx="466.5" cy="210" r="5" class="d-up-dot"/><text x="480.5" y="214">+13/βˆ’5 p=0.1</text></svg></div></div>
</section>
<section>
<h2>How it fails</h2>
<p class="takeaway">Two DAgger rounds cut <em>no_contact</em> from 57 to 24. <em>Holding</em> rose from 4 to 18 in round 2 and eased to 13 in round 3: the policy grasps but does not always let go on target.</p>
<p>Each bar is one checkpoint's failures, split by the label <code>FailureWatch</code> assigns. FLUX fails four times as often, mostly without touching the block.</p>
<div class="chart"><div class="scroll"><svg viewBox="0 0 680 418" width="100%" style="min-width:520px" role="img" aria-label="Failure labels as a share of each checkpoint's failures"><text x="112" y="25" text-anchor="end" class="lab">v5</text><rect x="120.0" y="10" width="181.7" height="22" class="f0"/><text x="210.9" y="25" text-anchor="middle" class="on">57</text><rect x="301.7" y="10" width="108.4" height="22" class="f1"/><text x="355.9" y="25" text-anchor="middle" class="on">34</text><rect x="410.1" y="10" width="76.5" height="22" class="f2"/><text x="448.4" y="25" text-anchor="middle" class="on">24</text><rect x="486.7" y="10" width="12.8" height="22" class="f3"/><rect x="499.4" y="10" width="25.5" height="22" class="f4"/><rect x="524.9" y="10" width="35.1" height="22" class="f5"/><text x="542.5" y="25" text-anchor="middle" class="on">11</text><text x="568" y="25" class="lab">138 failed</text><text x="112" y="59" text-anchor="end" class="lab">DAgger r1</text><rect x="120.0" y="44" width="141.4" height="22" class="f0"/><text x="190.7" y="59" text-anchor="middle" class="on">45</text><rect x="261.4" y="44" width="125.7" height="22" class="f1"/><text x="324.3" y="59" text-anchor="middle" class="on">40</text><rect x="387.1" y="44" width="103.7" height="22" class="f2"/><text x="439.0" y="59" text-anchor="middle" class="on">33</text><rect x="490.9" y="44" width="40.9" height="22" class="f3"/><text x="511.3" y="59" text-anchor="middle" class="on">13</text><rect x="531.7" y="44" width="28.3" height="22" class="f4"/><text x="545.9" y="59" text-anchor="middle" class="on">9</text><text x="568" y="59" class="lab">140 failed</text><text x="112" y="93" text-anchor="end" class="lab">DAgger r2</text><rect x="120.0" y="78" width="129.6" height="22" class="f0"/><text x="184.8" y="93" text-anchor="middle" class="on">33</text><rect x="249.6" y="78" width="121.8" height="22" class="f1"/><text x="310.5" y="93" text-anchor="middle" class="on">31</text><rect x="371.4" y="78" width="102.1" height="22" class="f2"/><text x="422.5" y="93" text-anchor="middle" class="on">26</text><rect x="473.6" y="78" width="70.7" height="22" class="f3"/><text x="508.9" y="93" text-anchor="middle" class="on">18</text><rect x="544.3" y="78" width="11.8" height="22" class="f4"/><rect x="556.1" y="78" width="3.9" height="22" class="f5"/><text x="568" y="93" class="lab">112 failed</text><text x="112" y="127" text-anchor="end" class="lab">DAgger r3</text><rect x="120.0" y="112" width="111.2" height="22" class="f0"/><text x="175.6" y="127" text-anchor="middle" class="on">24</text><rect x="231.2" y="112" width="148.2" height="22" class="f1"/><text x="305.3" y="127" text-anchor="middle" class="on">32</text><rect x="379.4" y="112" width="78.7" height="22" class="f2"/><text x="418.7" y="127" text-anchor="middle" class="on">17</text><rect x="458.1" y="112" width="60.2" height="22" class="f3"/><text x="488.2" y="127" text-anchor="middle" class="on">13</text><rect x="518.3" y="112" width="41.7" height="22" class="f4"/><text x="539.2" y="127" text-anchor="middle" class="on">9</text><text x="568" y="127" class="lab">95 failed</text><text x="112" y="161" text-anchor="end" class="lab">DAgger r3 retrain</text><rect x="120.0" y="146" width="85.2" height="22" class="f0"/><text x="162.6" y="161" text-anchor="middle" class="on">18</text><rect x="205.2" y="146" width="165.6" height="22" class="f1"/><text x="288.0" y="161" text-anchor="middle" class="on">35</text><rect x="370.8" y="146" width="104.1" height="22" class="f2"/><text x="422.8" y="161" text-anchor="middle" class="on">22</text><rect x="474.8" y="146" width="61.5" height="22" class="f3"/><text x="505.6" y="161" text-anchor="middle" class="on">13</text><rect x="536.3" y="146" width="23.7" height="22" class="f4"/><text x="568" y="161" class="lab">93 failed</text><text x="112" y="195" text-anchor="end" class="lab">DAgger r4</text><rect x="120.0" y="180" width="163.4" height="22" class="f0"/><text x="201.7" y="195" text-anchor="middle" class="on">39</text><rect x="283.4" y="180" width="79.6" height="22" class="f1"/><text x="323.2" y="195" text-anchor="middle" class="on">19</text><rect x="363.0" y="180" width="104.8" height="22" class="f2"/><text x="415.4" y="195" text-anchor="middle" class="on">25</text><rect x="467.8" y="180" width="71.2" height="22" class="f3"/><text x="503.4" y="195" text-anchor="middle" class="on">17</text><rect x="539.0" y="180" width="21.0" height="22" class="f4"/><text x="568" y="195" class="lab">105 failed</text><text x="112" y="229" text-anchor="end" class="lab">DAgger r5</text><rect x="120.0" y="214" width="101.8" height="22" class="f0"/><text x="170.9" y="229" text-anchor="middle" class="on">28</text><rect x="221.8" y="214" width="116.4" height="22" class="f1"/><text x="280.0" y="229" text-anchor="middle" class="on">32</text><rect x="338.2" y="214" width="141.8" height="22" class="f2"/><text x="409.1" y="229" text-anchor="middle" class="on">39</text><rect x="480.0" y="214" width="69.1" height="22" class="f3"/><text x="514.5" y="229" text-anchor="middle" class="on">19</text><rect x="549.1" y="214" width="7.3" height="22" class="f4"/><rect x="556.4" y="214" width="3.6" height="22" class="f5"/><text x="568" y="229" class="lab">121 failed</text><text x="112" y="263" text-anchor="end" class="lab">FLUX LoRA</text><rect x="120.0" y="248" width="299.5" height="22" class="f0"/><text x="269.7" y="263" text-anchor="middle" class="on">358</text><rect x="419.5" y="248" width="96.2" height="22" class="f1"/><text x="467.6" y="263" text-anchor="middle" class="on">115</text><rect x="515.7" y="248" width="18.4" height="22" class="f2"/><rect x="534.1" y="248" width="20.1" height="22" class="f3"/><rect x="554.1" y="248" width="5.9" height="22" class="f5"/><text x="568" y="263" class="lab">526 failed</text><text x="112" y="297" text-anchor="end" class="lab">v11 cotrain</text><rect x="120.0" y="282" width="131.2" height="22" class="f0"/><text x="185.6" y="297" text-anchor="middle" class="on">31</text><rect x="251.2" y="282" width="110.0" height="22" class="f1"/><text x="306.2" y="297" text-anchor="middle" class="on">26</text><rect x="361.2" y="282" width="101.5" height="22" class="f2"/><text x="411.9" y="297" text-anchor="middle" class="on">24</text><rect x="462.7" y="282" width="71.9" height="22" class="f3"/><text x="498.7" y="297" text-anchor="middle" class="on">17</text><rect x="534.6" y="282" width="25.4" height="22" class="f4"/><text x="568" y="297" class="lab">104 failed</text><text x="112" y="331" text-anchor="end" class="lab">Synthesis (r6)</text><rect x="120.0" y="316" width="110.8" height="22" class="f0"/><text x="175.4" y="331" text-anchor="middle" class="on">35</text><rect x="230.8" y="316" width="107.6" height="22" class="f1"/><text x="284.6" y="331" text-anchor="middle" class="on">34</text><rect x="338.4" y="316" width="104.5" height="22" class="f2"/><text x="390.6" y="331" text-anchor="middle" class="on">33</text><rect x="442.9" y="316" width="76.0" height="22" class="f3"/><text x="480.9" y="331" text-anchor="middle" class="on">24</text><rect x="518.8" y="316" width="38.0" height="22" class="f4"/><text x="537.8" y="331" text-anchor="middle" class="on">12</text><rect x="556.8" y="316" width="3.2" height="22" class="f5"/><text x="568" y="331" class="lab">139 failed</text><text x="112" y="365" text-anchor="end" class="lab">Autopilot 1</text><rect x="120.0" y="350" width="114.1" height="22" class="f0"/><text x="177.0" y="365" text-anchor="middle" class="on">35</text><rect x="234.1" y="350" width="110.8" height="22" class="f1"/><text x="289.5" y="365" text-anchor="middle" class="on">34</text><rect x="344.9" y="350" width="110.8" height="22" class="f2"/><text x="400.3" y="365" text-anchor="middle" class="on">34</text><rect x="455.7" y="350" width="68.4" height="22" class="f3"/><text x="489.9" y="365" text-anchor="middle" class="on">21</text><rect x="524.1" y="350" width="32.6" height="22" class="f4"/><text x="540.4" y="365" text-anchor="middle" class="on">10</text><rect x="556.7" y="350" width="3.3" height="22" class="f5"/><text x="568" y="365" class="lab">135 failed</text><rect x="120" y="394" width="10" height="10" class="f0"/><text x="134" y="403">no_contact</text><rect x="212" y="394" width="10" height="10" class="f1"/><text x="226" y="403">pushed</text><rect x="304" y="394" width="10" height="10" class="f2"/><text x="318" y="403">dropped</text><rect x="396" y="394" width="10" height="10" class="f3"/><text x="410" y="403">holding</text><rect x="488" y="394" width="10" height="10" class="f4"/><text x="502" y="403">fell_off</text><rect x="580" y="394" width="10" height="10" class="f5"/><text x="594" y="403">knocked_place</text></svg></div></div>
</section>
<section>
<h2>The DAgger loop</h2>
<p>Each round rolls the current policy out, snapshots each failure, keeps the most uncertain within a balanced budget, and lets the scripted expert record a recovery from that exact state. Recoveries are weighted Γ—8 in training from round 2 on.</p>
<pre class="mermaid">
flowchart LR
P["current policy"] -->|800 rollouts| F["failure snapshots"]
F -->|"uncertainty band 5–80%, budget per type"| K["kept states"]
K -->|scripted expert| R["recoveries"]
R -->|"Γ—8 + base demos"| T["fine-tune 4k steps"]
T --> B["6 suites Γ— 96 vs v5"]
B -->|"pooled gain, no suite worse"| P
</pre>
<p class="takeaway" style="margin-top:16px">Each round the policy fails less and the script rescues only about 40% of kept states, so recoveries shrank 85 β†’ 72 β†’ 59. Rounds 4 and 5 swapped in the residual RL expert, which recovered 88 of 120 and 72 of 113, but the policy got worse.</p>
<div class="chart"><div class="scroll"><svg viewBox="0 0 1120 206" width="100%" style="min-width:520px" role="img" aria-label="DAgger funnels per round"><text x="10" y="14" class="lab strong">Round 1 (v5)</text><rect x="10" y="26" width="190.0" height="20" class="fn0"/><text x="204.0" y="40" class="lab">800</text><rect x="10" y="56" width="108.3" height="20" class="fn1"/><text x="122.3" y="70" class="lab">456</text><rect x="10" y="86" width="78.4" height="20" class="fn2"/><text x="92.4" y="100" class="lab">330</text><rect x="10" y="116" width="49.2" height="20" class="fn3"/><text x="63.2" y="130" class="lab">207</text><rect x="10" y="146" width="20.2" height="20" class="fn4"/><text x="34.2" y="160" class="lab">85</text><text x="230" y="14" class="lab strong">Round 2 (v5)</text><rect x="230" y="26" width="190.0" height="20" class="fn0"/><text x="424.0" y="40" class="lab">800</text><rect x="230" y="56" width="117.3" height="20" class="fn1"/><text x="351.3" y="70" class="lab">494</text><rect x="230" y="86" width="70.3" height="20" class="fn2"/><text x="304.3" y="100" class="lab">296</text><rect x="230" y="116" width="44.9" height="20" class="fn3"/><text x="278.9" y="130" class="lab">189</text><rect x="230" y="146" width="17.1" height="20" class="fn4"/><text x="251.1" y="160" class="lab">72</text><text x="450" y="14" class="lab strong">Round 3 (r2)</text><rect x="450" y="26" width="190.0" height="20" class="fn0"/><text x="644.0" y="40" class="lab">800</text><rect x="450" y="56" width="136.3" height="20" class="fn1"/><text x="590.3" y="70" class="lab">574</text><rect x="450" y="86" width="46.8" height="20" class="fn2"/><text x="500.8" y="100" class="lab">197</text><rect x="450" y="116" width="34.7" height="20" class="fn3"/><text x="488.7" y="130" class="lab">146</text><rect x="450" y="146" width="14.0" height="20" class="fn4"/><text x="468.0" y="160" class="lab">59</text><text x="670" y="14" class="lab strong">Round 4 (r3, RL expert)</text><rect x="670" y="26" width="190.0" height="20" class="fn0"/><text x="864.0" y="40" class="lab">800</text><rect x="670" y="56" width="145.8" height="20" class="fn1"/><text x="819.8" y="70" class="lab">614</text><rect x="670" y="86" width="39.2" height="20" class="fn2"/><text x="713.2" y="100" class="lab">165</text><rect x="670" y="116" width="28.5" height="20" class="fn3"/><text x="702.5" y="130" class="lab">120</text><rect x="670" y="146" width="20.9" height="20" class="fn4"/><text x="694.9" y="160" class="lab">88</text><text x="890" y="14" class="lab strong">Round 5 (r3, RL x2)</text><rect x="890" y="26" width="190.0" height="20" class="fn0"/><text x="1084.0" y="40" class="lab">800</text><rect x="890" y="56" width="144.9" height="20" class="fn1"/><text x="1038.9" y="70" class="lab">610</text><rect x="890" y="86" width="38.0" height="20" class="fn2"/><text x="932.0" y="100" class="lab">160</text><rect x="890" y="116" width="26.8" height="20" class="fn3"/><text x="920.8" y="130" class="lab">113</text><rect x="890" y="146" width="17.1" height="20" class="fn4"/><text x="911.1" y="160" class="lab">72</text><rect x="10" y="186" width="10" height="10" class="fn0"/><text x="24" y="195">rollouts</text><rect x="138" y="186" width="10" height="10" class="fn1"/><text x="152" y="195">succeeded</text><rect x="266" y="186" width="10" height="10" class="fn2"/><text x="280" y="195">recoverable</text><rect x="394" y="186" width="10" height="10" class="fn3"/><text x="408" y="195">kept</text><rect x="522" y="186" width="10" height="10" class="fn4"/><text x="536" y="195">expert recovered</text></svg></div></div>
</section>
<section>
<h2>FLUX predicts well and acts badly</h2>
<p class="takeaway">Open-loop the LoRA beats holding the pose three to one at 32 steps; closed-loop it succeeds 50 times in 576.</p>
<p>Mean action error on 60 held-out demo frames, each fed its 7 preceding frames. The quantile bridge sits inside this loop both ways, so the mapping is not the problem.</p>
<div class="chart"><div class="scroll"><svg viewBox="0 0 640 226" width="100%" style="min-width:520px" role="img" aria-label="Open-loop action error by horizon"><line x1="50" y1="180" x2="610" y2="180" class="axis"/><text x="44" y="184.0" text-anchor="end">0</text><line x1="50" y1="180.0" x2="610" y2="180.0" class="grid"/><text x="44" y="104.0" text-anchor="end">4</text><line x1="50" y1="100.0" x2="610" y2="100.0" class="grid"/><text x="44" y="24.0" text-anchor="end">9</text><line x1="50" y1="20.0" x2="610" y2="20.0" class="grid"/><polyline points="67.5,175.9 190.0,160.8 330.0,148.9 610.0,131.8" class="l0" fill="none" stroke-width="2.5"/><circle cx="610.0" cy="131.8" r="4" class="l0-dot"/><rect x="60" y="196" width="12" height="4" class="l0-dot"/><text x="78" y="201">fair LoRA</text><polyline points="67.5,173.4 190.0,139.8 330.0,102.8 610.0,34.4" class="l1" fill="none" stroke-width="2.5"/><circle cx="610.0" cy="34.4" r="4" class="l1-dot"/><rect x="230" y="196" width="12" height="4" class="l1-dot"/><text x="248" y="201">zero-shot FLUX</text><polyline points="67.5,170.6 190.0,136.1 330.0,97.2 610.0,33.5" class="l2" fill="none" stroke-width="2.5"/><circle cx="610.0" cy="33.5" r="4" class="l2-dot"/><rect x="400" y="196" width="12" height="4" class="l2-dot"/><text x="418" y="201">hold pose</text><text x="610" y="218" text-anchor="end">steps ahead</text><text x="10" y="10">mean |action error|</text></svg></div></div>
</section>
<section>
<h2>An RL expert to recover what the script cannot</h2>
<p class="takeaway">A residual policy on top of the scripted expert recovers 74.0% of held-out failure snapshots; the expert alone recovers 45.9%.</p>
<p>Recovery from DAgger failure states the policy never trained on (W&amp;B run <code>fb6yv89f</code>), with fresh scenes still at 100%. Shaped PPO from scratch stays near 6% training success and is not shown. The next DAgger round records recoveries with this expert.</p>
<div class="chart"><div class="scroll"><svg viewBox="0 0 640 226" width="100%" style="min-width:520px" role="img" aria-label="RL expert held-out recovery"><line x1="50" y1="180" x2="610" y2="180" class="axis"/><text x="44" y="184.0" text-anchor="end">0</text><line x1="50" y1="180.0" x2="610" y2="180.0" class="grid"/><text x="44" y="104.0" text-anchor="end">50</text><line x1="50" y1="100.0" x2="610" y2="100.0" class="grid"/><text x="44" y="24.0" text-anchor="end">100</text><line x1="50" y1="20.0" x2="610" y2="20.0" class="grid"/><line x1="50" y1="106.6" x2="610" y2="106.6" class="ref"/><text x="610" y="100.6" text-anchor="end">scripted expert alone: 45.9%</text><polyline points="50.0,106.6 137.5,81.4 225.0,71.5 312.5,68.2 400.0,66.1 487.5,65.0 575.0,61.6" class="l0" fill="none" stroke-width="2.5"/><circle cx="575.0" cy="61.6" r="4" class="l0-dot"/><rect x="60" y="196" width="12" height="4" class="l0-dot"/><text x="78" y="201">residual PPO on the scripted expert</text><text x="610" y="218" text-anchor="end">thousand env steps</text><text x="10" y="10">held-out recovery %</text></svg></div></div>
</section>
<section>
<h2>Autopilot: the loop that trains on what Preflight refuses</h2>
<p class="takeaway">The first cycle taught a task the policy had never seen, 6 β†’ 86 of 96, and transferred nothing to the task it held out (ring, 0 β†’ 0). It also cost stacking (clean_many 88 β†’ 76, real_1_pomade_tin 87 β†’ 68), so the gate discarded it.</p>
<p>Every scene Preflight won't clear becomes a learning request. Autopilot ranks a fixed menu of tested recipes by the failures each could win back. Requests from real refusals rank first. Nemotron on Token Factory picks one of the top three; it never scores the result. The paired gate G3 judges it, and a human approves promotion. After cycle 1, Nemotron's proposed ring round was dropped as overtaken by the evidence. Cycle 4, a DAgger round answering four Preflight refusals, was discarded: no gain on new objects and both real-object suites worse. Live on the <a href="https://huggingface.co/spaces/chaleong/autopilot" target="_blank" rel="noopener">Autopilot Space</a>.</p>
<pre class="mermaid">
flowchart LR
PF["Preflight: RETRY / ASK_HUMAN"] -->|learning request| AP["Autopilot: rank menu"]
BM["benchmark of served policy"] --> AP
AP -->|top 3 eligible| N["Nemotron picks one"]
N --> T["train on the A100"]
T --> G{"G3: target gain, no suite worse"}
G -->|pass| H["human approves"] --> PF
G -->|fail| L["ledger: DISCARD / KEEP"]
</pre>
</section>
<section>
<h2>Every experiment, with its verdict</h2>
<div class="ledger">
<div class="row"><span class="pill promoted">Promoted</span><div><strong>DAgger round 2, recoveries Γ—8</strong>Pooled 464 vs 438, no suite worse. Served until round 3.</div></div>
<div class="row"><span class="pill flat">Kept</span><div><strong>DAgger round 1</strong>436 vs 438. Recoveries were 3.5% of training, too diluted to matter.</div></div>
<div class="row"><span class="pill promoted">Promoted</span><div><strong>DAgger round 3</strong>Retrained from the kept recoveries: 483 vs round 2's 464, pooled better, no suite worse; now served. The first run scored 481 with held_out_objects 61 (p=0.043); the retrain 57, above round 2 in both runs.</div></div>
<div class="row"><span class="pill flat">Kept</span><div><strong>DAgger round 4, RL expert records recoveries</strong>The residual RL expert recovered 88 of 120 kept failure states (73%, vs the script's 40%), but the candidate scored 471 vs 483 (within noise) and no_contact rose from 18 to 39.</div></div>
<div class="row"><span class="pill rejected">Rejected</span><div><strong>DAgger round 5, RL recoveries weighted Γ—2</strong>72 of 113 recovered; 160 RL recoveries at Γ—2 beside the scripted Γ—8. 455 vs 483 and held_out_objects 41 vs 57 (p=0.004): the RL-recovery line is closed.</div></div>
<div class="row"><span class="pill rejected">Rejected</span><div><strong>FLUX 3 Action LoRA (fair)</strong>Same 800 demos, same sample volume: 50 vs 438, worse on every suite (p &lt; 0.001), 0/7 externally. Executing 16 instead of 32 actions per chunk did not help.</div></div>
<div class="row"><span class="pill flat">Kept</span><div><strong>FLUX, current frame + absolute targets</strong>Same demos at a quarter of the budget, action heads re-initialised: held_out_objects 21 vs 3 (p&lt;0.001), clean_many 15 vs 9. The self-history and deltas caused most of the collapse; still far below round 3.</div></div>
<div class="row"><span class="pill flat">Flat</span><div><strong>Targeted demos (v6)</strong>Staged-mistake recoveries plus lookalike decoys: 421 vs 438, within noise.</div></div>
<div class="row"><span class="pill rejected">Rejected</span><div><strong>Visual domain randomisation (v7)</strong>Lifts the Seeed wrist-mount scenes 15 β†’ 53 but drops clean_many 81 β†’ 58; 1/50 on so101-benchmark.</div></div>
<div class="row"><span class="pill flat">Kept</span><div><strong>RL expert (residual PPO)</strong>74.0% held-out snapshot recovery vs the scripted expert's 45.9%. Plain PPO stopped at 1% after its reward paid hovering; shaped PPO from scratch stays near 6%.</div></div>
<div class="row"><span class="pill flat">Flat</span><div><strong>Best-of-8 chunk consensus</strong>Medoid of 8 flow-matching samples on round 2: held_out_objects 55 vs 50 (p=0.41), clean_many 87 vs 86. Within noise at 8Γ— inference cost.</div></div>
<div class="row"><span class="pill flat">Flat</span><div><strong>DiVeR verifier, best-of-8 on round 3</strong>Trained on 300 round-3 rollouts, it ranks held-out steps at AUC 0.79, yet picking with it scores held_out_objects 54 vs 57 (p=0.66) and clean_many 85 vs 88. Best-of-N is closed.</div></div>
<div class="row"><span class="pill rejected">Rejected</span><div><strong>DiVeR on the action expert's per-candidate state</strong>Each candidate scored by the action head's own last denoising output: AUC 0.77, held_out_objects 48 vs 57 (p=0.049, worse).</div></div>
<div class="row"><span class="pill flat">Flat</span><div><strong>DiVeR with real ArmnetBench outcomes</strong>Plus 305 calibrated real block-stacking episodes (87 successful, 218 failed): AUC 0.77, held_out_objects 52 vs 57, clean_many 83 vs 88.</div></div>
<div class="row"><span class="pill rejected">Rejected</span><div><strong>Real-data co-training (v9)</strong>400 ArmnetBench SO-101 teleop demos of new objects at 23% of samples: 373 vs 438, held_out_objects 48 vs 52 (no gain), significantly worse on three suites, 0/50 externally.</div></div>
<div class="row"><span class="pill flat">Kept</span><div><strong>Calibrated co-training (v11)</strong>Joint offsets fitted to the twin cut real frames under the table from 99.7% to 4.2%. Round 3 with 87 calibrated block-stacking episodes at 9%: 472 vs 483, every suite within noise. Neutral in the twin; kept for the real arm.</div></div>
<div class="row"><span class="pill rejected">Rejected</span><div><strong>Failure-state synthesis (round 6)</strong>Expert recoveries recorded from jittered failure states: 437 vs 483, worse on held_out_objects (43), clean_many (76) and real_1_pomade_tin (75). The policy does not need recoveries from states it never reaches.</div></div>
<div class="row"><span class="pill rejected">Rejected</span><div><strong>Autopilot cycle 1: multi-task round, ring held out</strong>400 basket demos in 5 phrasings: basket 6 β†’ 86/96 (+82/βˆ’2), ring 0 β†’ 0. Stacking 441 vs 483, clean_many and real_1_pomade_tin significantly worse. Gate G3 discarded it.</div></div>
<div class="row"><span class="pill rejected">Rejected</span><div><strong>Autopilot cycle 4: DAgger round 7, answering Preflight's refusals</strong>Chosen by Nemotron for the objects and environments axes: 32 of 123 kept failure states recovered (26%). 461 vs 483; held_out_objects 59 (within noise) but both pomade-tin suites significantly worse. DAgger is near saturation on these failure modes.</div></div>
</div>
</section>
<section>
<h2>What is not established</h2>
<ul class="plain">
<li>Nothing has run on the physical arm. The repo will be updated with live results when it arrives.</li>
<li>Round 3's new-object gain (p=0.043) is one suite of six, uncorrected for multiple comparisons, from one seed.</li>
<li>No single suite gain from round 3's retrain is significant on its own; the pooled gain passed a gate fixed in advance, twice in a row for rounds 2 and 3.</li>
<li>Which of FLUX's two changes (current-frame conditioning, absolute targets) carries its 21-vs-3 gain; they were tested together.</li>
<li>The RL expert is measured on recovering failure snapshots, not yet as a source of DAgger data.</li>
<li>Whether a smaller share of new-task demos keeps the basket gain without the stacking loss; cycle 1 tried one mix (demos Γ—2, about 20% of episodes).</li>
<li>Nemotron's choices are logged, not validated: the gate judges each cycle's result, not whether another recipe would have done better.</li>
<li>One benchmark seed and one training seed per checkpoint.</li>
</ul>
</section>
<footer>
Sources: <a href="https://github.com/charleneleong-ai/nebius-physical-ai" target="_blank" rel="noopener">charleneleong-ai/nebius-physical-ai</a> β€” <code>docs/report.pdf</code>, <code>docs/experiment-card.md</code>, <code>docs/generalisation-diagnosis.md</code>; rebuilt by <code>scripts/build_artifact.py</code>. Runs: <a href="https://wandb.ai/chaleong/preflight/reports/Preflight:-SO-101-generalisation-experiments--VmlldzoxODA5MDQ3Mg==" target="_blank" rel="noopener">W&amp;B report</a>.
</footer>
</main>
<script type="module">
import mermaid from "https://cdn.jsdelivr.net/npm/mermaid@11.4.1/dist/mermaid.esm.min.mjs";
mermaid.initialize({ startOnLoad: true, theme: "dark" });
</script>
</body>
</html>