# Tactile feature preprocessing — T3 / SITR vs HTT (what happens to a raw frame)

`written: 2026-08-12` · applies to the screw-task ACT policies and their
feature encoders (`tools/encode_t3.py`, `tools/encode_sitr.py`,
`tools/live_features.py`; HTT = the in-graph `tf9dpe` convention).

## Short answer

**T3 and SITR consume RAW tactile images — no reference / background-frame
subtraction.** Each follows its authors' published input convention. Only the
HTT features use reference subtraction.

## Operation chains (from `/images/raw_<f>`: 240×320 uint8 grayscale)

### T3 (matches the paper's T3 probe rows exactly)

1. uint8 → float, ÷255 → [0,1]
2. grayscale replicated to 3 channels
3. bilinear resize → 224×224
4. normalize `(x − 0.49611) / 0.18685` — **per-dataset stats computed once on
   the screwing corpus** (T3's convention is per-dataset stats on raw RGB;
   borrowing another dataset's stats was measured to put inputs several σ
   off-distribution during the tactile_fusion probe work). These constants are
   training-fixed — never recompute at deployment.
5. → T3-medium encoder (domain `mini`) → 9-block trunk → **non-affine**
   LayerNorm → cls token → 768-d per finger (×5 fingers, order 0..4 → 3840-d).

### SITR

1–3. same ÷255 → 3-channel replicate → 224×224
4. normalize with SITR's training stats applied on the [0,1] tensor:
   mean = (−1.2223, −1.8114, −1.7090), std = (11.7932, 12.7956, 13.6452)
   (from `gsrl/dataloaders.py`).
5. → `SITR_base(num_calibration=0).forward_encoder(x, c=None)`, cls token →
   768-d per finger. **`num_calibration=0`**: SITR's sensor-invariance
   mechanism normally consumes calibration frames (its own flavor of reference
   frames; the tactile_fusion probe baselines used 18) — the ACT feature
   encoding runs it calibration-free.

### HTT (`tf9dpe`, for contrast — the only one with reference frames)

1. per-episode **background frame subtraction** (the `tactile_bg_224` group;
   at deployment: captured at the settled episode-start pose)
2. division by image std
3. **2-frame temporal stacking** (`tactile_image_pair`)
4. resize 224×224 → 9dtact ViT encoder + shared trunk → 192-d per finger
   (×5 → 960-d).

## Why the asymmetry is deliberate

Each encoder receives its authors' released input convention — the defensible
way to run baselines. Imposing HTT's bg-subtraction on T3/SITR would deviate
from their published usage and invite a "you ran them wrong" objection.

## Practical implication for robot deployment

T3/SITR policies have **no reference-frame mechanism at all**: if the
fingertip cameras' resting appearance drifts between the training data and
the rig (lighting, gel wear, mounting), their features shift with nothing to
anchor them. Interpret on-robot behavior differences with this in mind. The
episode-start background capture fix (see `real_deploy_script/DEPLOY_FIXES.md`)
matters only for HTT — T3/SITR need no bg capture, by design.