File size: 4,122 Bytes
121869d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | # For the Claude on the target workstation — updating your inference code
`written: 2026-08-11` · applies to the bundle in
`hf.co/AllenBi21/tactile-act-screw-baselines`
This repo contains the **reference inference stack** the `screw_t3_50k` /
`screw_sitr5_50k` checkpoints were validated with: `inference.py`,
`act_policy/`, `data/dataset.py`, `tools/live_features.py`. Your local
inference code is known to have drifted slightly. **Update yours against this
reference — do not wholesale-replace it** (your local robot-IO, rate control,
logging, and device plumbing should survive), with one exception noted in §3.
## 1. Procedure
1. Download: `hf download AllenBi21/tactile-act-screw-baselines --local-dir ref`
2. Diff `ref/inference.py` and `ref/act_policy/` against your local versions.
3. Port every difference that touches the load-bearing contract (§2) from the
reference into your code. Keep your local plumbing everywhere else.
4. Run the verification in §4 before any robot use. Do not skip it.
## 2. Load-bearing contract (must match the reference exactly)
1. **Checkpoint loading**: the .pt is a dict
`{"step", "model", "config", "optimizer", ...}`; build `ACTConfig` from
`config`, `load_state_dict(ckpt["model"])` must report
**missing=[] unexpected=[]**. qpos/tactile/action normalization lives in
buffers inside the state_dict — never re-normalize inputs externally.
2. **Observation contract**: `{"state": (22,) float, "tactile": (3840,) float}`.
The tactile vector is 5 fingers × 768, **concatenated in finger order
0..4**, produced by `tools/live_features.py` (its preprocessing constants
are training-fixed — do not "improve" or recompute them).
3. **Chunking**: `chunk_size=30, n_action_steps=30` come from the checkpoint
config; `step()` pops from the action queue and re-queries every 30 frames;
`reset()` must be called at every episode start.
4. **Output semantics**: `step()`/`predict_chunk()` return **absolute target
joint positions** (22,), even for delta-trained checkpoints (integration
happens inside `policy.predict_action_chunk`).
5. `state_history = tactile_history = 1` for these checkpoints.
## 3. When to wholesale-replace instead
If your local `act_policy/` predates the May refactor (no `forward_context`
split, no `FlowMatchingHead`, no generalized tactile-key routing), take the
bundled `act_policy/` as-is — partial merges of the model code are how silent
weight-mismatch bugs happen. The gate is §2.1: if `load_state_dict` is not
clean, replace the package.
## 4. Verification (in order, all must pass)
1. **Feature parity** (needs any episode HDF5 that contains stored `/t3/<f>`
and `/sitr/<f>` datasets):
```bash
python -m tools.live_features --kind t3 --episode <episode.hdf5>
python -m tools.live_features --kind sitr --episode <episode.hdf5>
# must print PASS (worst cosine > 0.999)
```
2. **Clean checkpoint load**: load both act_best.pt; assert no missing /
unexpected keys, and `config.tactile_dim == 3840`.
3. **Offline eval sanity** (if the episode data is present):
```bash
python inference.py --checkpoint <...>/screw_t3_50k/act_best.pt \
--mode offline --data episodes/screwing \
--tactile-keys t3_0 t3_1 t3_2 t3_3 t3_4 --out-csv /tmp/t3.csv
```
Expect overall L1 ≈ 0.023 / val-episode L1 ≈ 0.032 (T3); ≈ 0.020 / 0.031
(SITR). A large deviation means a contract violation — stop and compare
against the reference again.
4. Only after 1–3 pass: dry-run the robot loop with `step()` on live data.
## 5. Environment notes
- `tools/live_features.py` needs the third-party encoder stacks:
tactile_fusion repo (`TACTILE_FUSION_ROOT` env; loads T3 via its
`load_baselines`) + t3 repo + t3_medium weights; gsrl repo (`GSRL_ROOT`) +
`checkpoints/SITR_B18.pth`. These are NOT in this bundle.
- Known import trap: tactile_act's `data/` package shadows tactile_fusion's;
`live_features.py` handles it (`_strip_shadowing_paths`) — keep that if you
port the file.
- torch ≥ 2.x; GPU strongly recommended for the SITR encoder (86M) at 30 Hz.
|