doc/verification.md

Verification

What actually works

Honest status, regenerated from the run logs — not a wish list. A task counts as passing only when its numeric conditions pass and the rendered contact sheet was inspected and agrees.

The numeric checks are deliberately strict, and three of them were caught lying during development (see DIAGNOSTICS.md): a pour that reported success with a failed grasp, and two joints that "moved" on gravity alone. Each was tightened rather than worked around, so several tasks moved from green to red on purpose.

Run python scripts/status.py to regenerate the table below from outputs/tasks/*.log.

Agent-in-the-loop verification

The numeric check only inspects final object poses. It will happily pass an episode where the cup landed upside-down, a gripper hung empty in the air, or a vessel was carried over the target without ever tipping — all three of those actually happened here. So every task is also reviewed visually, and only numeric=SUCCESS and visual=CONFIRMED counts as done.

python scripts/contact_sheet.py outputs/tasks/*.mp4   # episode -> one strip of frames
python scripts/yam_agent_loop.py evaluate             # contact sheets + per-task review prompts
python scripts/yam_agent_loop.py update               # verdicts -> concrete parameter fixes
python scripts/yam_agent_loop.py propose              # uncovered skill axes from the asset library

evaluate renders a sheet per episode and emits a prompt naming what to look for in that specific task (is the cup upright? is anything actually in the gripper? did the lid come off before the food went in?). update maps a verdict onto a fix through a symptom→cause table built from the failures in DIAGNOSTICS.md — e.g. "jaws closed on nothing" → measure the mesh origin offset, not the clamp force. propose reads the converted asset library and suggests skills not yet covered.

Reviewing this way is what caught the three false positives described above; the fix in each case was to tighten the numeric condition, so the visual pass and the numeric pass now agree.