Hugging Face's logo

AllenBi21
/
tactile-act-screw-baselines

Model card Files
xet
 Community
tactile-act-screw-baselines / FEATURE_PREPROCESSING.md
AllenBi21's picture
AllenBi21
Add feature-preprocessing doc
bd86863 verified
preview code
|
Raw History Blame Contribute Delete
3 kB

Tactile feature preprocessing β€” T3 / SITR vs HTT (what happens to a raw frame)

written: 2026-08-12 Β· applies to the screw-task ACT policies and their feature encoders (tools/encode_t3.py, tools/encode_sitr.py, tools/live_features.py; HTT = the in-graph tf9dpe convention).

Short answer

T3 and SITR consume RAW tactile images β€” no reference / background-frame subtraction. Each follows its authors' published input convention. Only the HTT features use reference subtraction.

Operation chains (from /images/raw_<f>: 240Γ—320 uint8 grayscale)

T3 (matches the paper's T3 probe rows exactly)

  1. uint8 β†’ float, Γ·255 β†’ [0,1]
  2. grayscale replicated to 3 channels
  3. bilinear resize β†’ 224Γ—224
  4. normalize (x βˆ’ 0.49611) / 0.18685 β€” per-dataset stats computed once on the screwing corpus (T3's convention is per-dataset stats on raw RGB; borrowing another dataset's stats was measured to put inputs several Οƒ off-distribution during the tactile_fusion probe work). These constants are training-fixed β€” never recompute at deployment.
  5. β†’ T3-medium encoder (domain mini) β†’ 9-block trunk β†’ non-affine LayerNorm β†’ cls token β†’ 768-d per finger (Γ—5 fingers, order 0..4 β†’ 3840-d).

SITR

1–3. same Γ·255 β†’ 3-channel replicate β†’ 224Γ—224 4. normalize with SITR's training stats applied on the [0,1] tensor: mean = (βˆ’1.2223, βˆ’1.8114, βˆ’1.7090), std = (11.7932, 12.7956, 13.6452) (from gsrl/dataloaders.py). 5. β†’ SITR_base(num_calibration=0).forward_encoder(x, c=None), cls token β†’ 768-d per finger. num_calibration=0: SITR's sensor-invariance mechanism normally consumes calibration frames (its own flavor of reference frames; the tactile_fusion probe baselines used 18) β€” the ACT feature encoding runs it calibration-free.

HTT (tf9dpe, for contrast β€” the only one with reference frames)

  1. per-episode background frame subtraction (the tactile_bg_224 group; at deployment: captured at the settled episode-start pose)
  2. division by image std
  3. 2-frame temporal stacking (tactile_image_pair)
  4. resize 224Γ—224 β†’ 9dtact ViT encoder + shared trunk β†’ 192-d per finger (Γ—5 β†’ 960-d).

Why the asymmetry is deliberate

Each encoder receives its authors' released input convention β€” the defensible way to run baselines. Imposing HTT's bg-subtraction on T3/SITR would deviate from their published usage and invite a "you ran them wrong" objection.

Practical implication for robot deployment

T3/SITR policies have no reference-frame mechanism at all: if the fingertip cameras' resting appearance drifts between the training data and the rig (lighting, gel wear, mounting), their features shift with nothing to anchor them. Interpret on-robot behavior differences with this in mind. The episode-start background capture fix (see real_deploy_script/DEPLOY_FIXES.md) matters only for HTT β€” T3/SITR need no bg capture, by design.