| # Tactile feature preprocessing β T3 / SITR vs HTT (what happens to a raw frame) |
|
|
| `written: 2026-08-12` Β· applies to the screw-task ACT policies and their |
| feature encoders (`tools/encode_t3.py`, `tools/encode_sitr.py`, |
| `tools/live_features.py`; HTT = the in-graph `tf9dpe` convention). |
|
|
| ## Short answer |
|
|
| **T3 and SITR consume RAW tactile images β no reference / background-frame |
| subtraction.** Each follows its authors' published input convention. Only the |
| HTT features use reference subtraction. |
|
|
| ## Operation chains (from `/images/raw_<f>`: 240Γ320 uint8 grayscale) |
| |
| ### T3 (matches the paper's T3 probe rows exactly) |
| |
| 1. uint8 β float, Γ·255 β [0,1] |
| 2. grayscale replicated to 3 channels |
| 3. bilinear resize β 224Γ224 |
| 4. normalize `(x β 0.49611) / 0.18685` β **per-dataset stats computed once on |
| the screwing corpus** (T3's convention is per-dataset stats on raw RGB; |
| borrowing another dataset's stats was measured to put inputs several Ο |
| off-distribution during the tactile_fusion probe work). These constants are |
| training-fixed β never recompute at deployment. |
| 5. β T3-medium encoder (domain `mini`) β 9-block trunk β **non-affine** |
| LayerNorm β cls token β 768-d per finger (Γ5 fingers, order 0..4 β 3840-d). |
|
|
| ### SITR |
|
|
| 1β3. same Γ·255 β 3-channel replicate β 224Γ224 |
| 4. normalize with SITR's training stats applied on the [0,1] tensor: |
| mean = (β1.2223, β1.8114, β1.7090), std = (11.7932, 12.7956, 13.6452) |
| (from `gsrl/dataloaders.py`). |
| 5. β `SITR_base(num_calibration=0).forward_encoder(x, c=None)`, cls token β |
| 768-d per finger. **`num_calibration=0`**: SITR's sensor-invariance |
| mechanism normally consumes calibration frames (its own flavor of reference |
| frames; the tactile_fusion probe baselines used 18) β the ACT feature |
| encoding runs it calibration-free. |
| |
| ### HTT (`tf9dpe`, for contrast β the only one with reference frames) |
| |
| 1. per-episode **background frame subtraction** (the `tactile_bg_224` group; |
| at deployment: captured at the settled episode-start pose) |
| 2. division by image std |
| 3. **2-frame temporal stacking** (`tactile_image_pair`) |
| 4. resize 224Γ224 β 9dtact ViT encoder + shared trunk β 192-d per finger |
| (Γ5 β 960-d). |
|
|
| ## Why the asymmetry is deliberate |
|
|
| Each encoder receives its authors' released input convention β the defensible |
| way to run baselines. Imposing HTT's bg-subtraction on T3/SITR would deviate |
| from their published usage and invite a "you ran them wrong" objection. |
|
|
| ## Practical implication for robot deployment |
|
|
| T3/SITR policies have **no reference-frame mechanism at all**: if the |
| fingertip cameras' resting appearance drifts between the training data and |
| the rig (lighting, gel wear, mounting), their features shift with nothing to |
| anchor them. Interpret on-robot behavior differences with this in mind. The |
| episode-start background capture fix (see `real_deploy_script/DEPLOY_FIXES.md`) |
| matters only for HTT β T3/SITR need no bg capture, by design. |
|
|
