doc/verification.md

# Verification

## What actually works

Honest status, regenerated from the run logs — not a wish list. A task counts as **passing** only
when its numeric conditions pass **and** the rendered contact sheet was inspected and agrees.

The numeric checks are deliberately strict, and three of them were caught *lying* during
development (see [`DIAGNOSTICS.md`](DIAGNOSTICS.md)): a pour that reported success with a failed
grasp, and two joints that "moved" on gravity alone. Each was tightened rather than worked
around, so several tasks moved from green to red on purpose.

Run `python scripts/status.py` to regenerate the table below from `outputs/tasks/*.log`.

### Agent-in-the-loop verification

The numeric check only inspects final object poses. It will happily pass an episode where the cup
landed upside-down, a gripper hung empty in the air, or a vessel was carried over the target
without ever tipping — all three of those actually happened here. So every task is also reviewed
visually, and only `numeric=SUCCESS` **and** `visual=CONFIRMED` counts as done.

```bash
python scripts/contact_sheet.py outputs/tasks/*.mp4   # episode -> one strip of frames
python scripts/yam_agent_loop.py evaluate             # contact sheets + per-task review prompts
python scripts/yam_agent_loop.py update               # verdicts -> concrete parameter fixes
python scripts/yam_agent_loop.py propose              # uncovered skill axes from the asset library
```

`evaluate` renders a sheet per episode and emits a prompt naming what to look for in that
specific task (is the cup upright? is anything actually in the gripper? did the lid come off
*before* the food went in?). `update` maps a verdict onto a fix through a symptom→cause table
built from the failures in [`DIAGNOSTICS.md`](DIAGNOSTICS.md) — e.g. *"jaws closed on nothing"* →
measure the mesh origin offset, not the clamp force. `propose` reads the converted asset library
and suggests skills not yet covered.

Reviewing this way is what caught the three false positives described above; the fix in each case
was to tighten the numeric condition, so the visual pass and the numeric pass now agree.