| # Verification |
|
|
| ## What actually works |
|
|
| Honest status, regenerated from the run logs — not a wish list. A task counts as **passing** only |
| when its numeric conditions pass **and** the rendered contact sheet was inspected and agrees. |
|
|
| The numeric checks are deliberately strict, and three of them were caught *lying* during |
| development (see [`DIAGNOSTICS.md`](DIAGNOSTICS.md)): a pour that reported success with a failed |
| grasp, and two joints that "moved" on gravity alone. Each was tightened rather than worked |
| around, so several tasks moved from green to red on purpose. |
|
|
| Run `python scripts/status.py` to regenerate the table below from `outputs/tasks/*.log`. |
|
|
| ### Agent-in-the-loop verification |
|
|
| The numeric check only inspects final object poses. It will happily pass an episode where the cup |
| landed upside-down, a gripper hung empty in the air, or a vessel was carried over the target |
| without ever tipping — all three of those actually happened here. So every task is also reviewed |
| visually, and only `numeric=SUCCESS` **and** `visual=CONFIRMED` counts as done. |
|
|
| ```bash |
| python scripts/contact_sheet.py outputs/tasks/*.mp4 # episode -> one strip of frames |
| python scripts/yam_agent_loop.py evaluate # contact sheets + per-task review prompts |
| python scripts/yam_agent_loop.py update # verdicts -> concrete parameter fixes |
| python scripts/yam_agent_loop.py propose # uncovered skill axes from the asset library |
| ``` |
|
|
| `evaluate` renders a sheet per episode and emits a prompt naming what to look for in that |
| specific task (is the cup upright? is anything actually in the gripper? did the lid come off |
| *before* the food went in?). `update` maps a verdict onto a fix through a symptom→cause table |
| built from the failures in [`DIAGNOSTICS.md`](DIAGNOSTICS.md) — e.g. *"jaws closed on nothing"* → |
| measure the mesh origin offset, not the clamp force. `propose` reads the converted asset library |
| and suggests skills not yet covered. |
|
|
| Reviewing this way is what caught the three false positives described above; the fix in each case |
| was to tighten the numeric condition, so the visual pass and the numeric pass now agree. |
|
|
|
|
