| # For the Claude on the target workstation — updating your inference code |
|
|
| `written: 2026-08-11` · applies to the bundle in |
| `hf.co/AllenBi21/tactile-act-screw-baselines` |
|
|
| This repo contains the **reference inference stack** the `screw_t3_50k` / |
| `screw_sitr5_50k` checkpoints were validated with: `inference.py`, |
| `act_policy/`, `data/dataset.py`, `tools/live_features.py`. Your local |
| inference code is known to have drifted slightly. **Update yours against this |
| reference — do not wholesale-replace it** (your local robot-IO, rate control, |
| logging, and device plumbing should survive), with one exception noted in §3. |
|
|
| ## 1. Procedure |
|
|
| 1. Download: `hf download AllenBi21/tactile-act-screw-baselines --local-dir ref` |
| 2. Diff `ref/inference.py` and `ref/act_policy/` against your local versions. |
| 3. Port every difference that touches the load-bearing contract (§2) from the |
| reference into your code. Keep your local plumbing everywhere else. |
| 4. Run the verification in §4 before any robot use. Do not skip it. |
|
|
| ## 2. Load-bearing contract (must match the reference exactly) |
|
|
| 1. **Checkpoint loading**: the .pt is a dict |
| `{"step", "model", "config", "optimizer", ...}`; build `ACTConfig` from |
| `config`, `load_state_dict(ckpt["model"])` must report |
| **missing=[] unexpected=[]**. qpos/tactile/action normalization lives in |
| buffers inside the state_dict — never re-normalize inputs externally. |
| 2. **Observation contract**: `{"state": (22,) float, "tactile": (3840,) float}`. |
| The tactile vector is 5 fingers × 768, **concatenated in finger order |
| 0..4**, produced by `tools/live_features.py` (its preprocessing constants |
| are training-fixed — do not "improve" or recompute them). |
| 3. **Chunking**: `chunk_size=30, n_action_steps=30` come from the checkpoint |
| config; `step()` pops from the action queue and re-queries every 30 frames; |
| `reset()` must be called at every episode start. |
| 4. **Output semantics**: `step()`/`predict_chunk()` return **absolute target |
| joint positions** (22,), even for delta-trained checkpoints (integration |
| happens inside `policy.predict_action_chunk`). |
| 5. `state_history = tactile_history = 1` for these checkpoints. |
|
|
| ## 3. When to wholesale-replace instead |
|
|
| If your local `act_policy/` predates the May refactor (no `forward_context` |
| split, no `FlowMatchingHead`, no generalized tactile-key routing), take the |
| bundled `act_policy/` as-is — partial merges of the model code are how silent |
| weight-mismatch bugs happen. The gate is §2.1: if `load_state_dict` is not |
| clean, replace the package. |
|
|
| ## 4. Verification (in order, all must pass) |
|
|
| 1. **Feature parity** (needs any episode HDF5 that contains stored `/t3/<f>` |
| and `/sitr/<f>` datasets): |
| ```bash |
| python -m tools.live_features --kind t3 --episode <episode.hdf5> |
| python -m tools.live_features --kind sitr --episode <episode.hdf5> |
| # must print PASS (worst cosine > 0.999) |
| ``` |
| 2. **Clean checkpoint load**: load both act_best.pt; assert no missing / |
| unexpected keys, and `config.tactile_dim == 3840`. |
| 3. **Offline eval sanity** (if the episode data is present): |
| ```bash |
| python inference.py --checkpoint <...>/screw_t3_50k/act_best.pt \ |
| --mode offline --data episodes/screwing \ |
| --tactile-keys t3_0 t3_1 t3_2 t3_3 t3_4 --out-csv /tmp/t3.csv |
| ``` |
| Expect overall L1 ≈ 0.023 / val-episode L1 ≈ 0.032 (T3); ≈ 0.020 / 0.031 |
| (SITR). A large deviation means a contract violation — stop and compare |
| against the reference again. |
| 4. Only after 1–3 pass: dry-run the robot loop with `step()` on live data. |
|
|
| ## 5. Environment notes |
|
|
| - `tools/live_features.py` needs the third-party encoder stacks: |
| tactile_fusion repo (`TACTILE_FUSION_ROOT` env; loads T3 via its |
| `load_baselines`) + t3 repo + t3_medium weights; gsrl repo (`GSRL_ROOT`) + |
| `checkpoints/SITR_B18.pth`. These are NOT in this bundle. |
| - Known import trap: tactile_act's `data/` package shadows tactile_fusion's; |
| `live_features.py` handles it (`_strip_shadowing_paths`) — keep that if you |
| port the file. |
| - torch ≥ 2.x; GPU strongly recommended for the SITR encoder (86M) at 30 Hz. |
|
|
