| # Real-robot deploy — findings & required fixes (2026-08-11) |
|
|
| Symptom investigated: qpos and wrench policies behave normally on hardware; |
| HTT (in-graph), T3, and SITR policies do not. |
|
|
| ## A. T3 / SITR — root cause found, new runner provided |
|
|
| `run_ACT.py` feeds `obs["tactile"] = wrench_1` (30-d wrench) to EVERY policy. |
| The T3/SITR checkpoints expect the 3840-d encoder feature vector; no encoder |
| exists anywhere in the deploy scripts. **Use the new `run_ACT_t3_sitr.py`**, |
| which computes features live via `tools/live_features.py` (validated against |
| the training features). Needs `TACTILE_FUSION_ROOT`/`GSRL_ROOT` env vars and |
| the encoder weights (see SETUP_ENCODERS_REMOTE.md in the HF bundle). |
| |
| ## B. HTT in-graph (`run_ACT_tactile_checkpoint.py`) — 3 line-level fixes |
| |
| 1. **Backgrounds captured at the wrong pose/time (primary suspect).** |
| Current order: command `hand_initial_pose` → sleep(1) → `set_backgrounds` |
| (l.130) → 'c' → move to `pose.json["pose1"]` → run. Training bg |
| (`tactile_bg_224`) = resting frame AT the episode start configuration. |
| FIX: move the two bg lines (l.130–131) to AFTER the `pose.json` move and |
| its `rospy.sleep(5)` (i.e., after l.144), so bg is captured settled, at the |
| pose the episode actually starts from, immediately before stepping: |
| ```python |
| rospy.sleep(5) |
| bg = {key: finger_image(sharpa_tactile[i]) for i, key in enumerate(FINGER_KEYS)} |
| infer.set_backgrounds(bg) |
| ``` |
| Also verify the fingers are NOT in contact at that pose — bg must be a |
| free/rest frame, as in the dataset. |
|
|
| 2. **`inference_tactile_checkpoint.py` l.112: `act_cfg["n_action_steps"] = 15`** |
| hard-overrides the trained value (30). Delete the line (keep the |
| `setdefault` above it) unless the 15-step re-query was a deliberate, |
| validated choice. |
| |
| 3. **Control rate: `rospy.Rate(50)` → `rospy.Rate(30)`** in both runners. |
| Demos are ~31 Hz; at 50 Hz chunks execute 1.6× too fast AND the in-graph |
| branch's rolling frame-pair history gets 20 ms spacing vs ~32 ms at |
| training — the image-history branch is the component most sensitive to |
| this. (qpos/wrench tolerating 50 Hz does not mean the HTT branch does.) |
| |
| ## C. Hygiene (not the bug, but will bite eventually) |
| |
| - `run_ACT.py` l.41 hardcodes `screw_qpos_50k` while `policies[arg]` suggests |
| selection — the log filename (l.52) records `policies[3]` regardless of |
| what actually ran. Make the checkpoint path derive from the selection. |
| - `finger_image`'s `arr.max() <= 1.5` float heuristic mis-scales a genuinely |
| dark uint8-as-float frame; prefer an explicit dtype contract from the |
| publisher. |
| |
| ## Recommended bring-up order after fixes |
| |
| 1. T3/SITR: run the `live_features` self-test on the robot machine first |
| (PASS required), then `run_ACT_t3_sitr.py` with the hand in free space — |
| confirm actions look like the demo's approach motion before touching the |
| object. |
| 2. HTT: after fixes B1–B3, log `policy`'s tactile-branch feature norm for a |
| few seconds at rest and compare to a training episode's feature norms — |
| a mismatch >2× means bg is still wrong. |
| |
