Hugging Face's logo

AllenBi21
/
tactile-act-screw-baselines

Model card Files
xet
 Community
tactile-act-screw-baselines
File size: 2,999 Bytes
bd86863
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# Tactile feature preprocessing β€” T3 / SITR vs HTT (what happens to a raw frame)

`written: 2026-08-12` Β· applies to the screw-task ACT policies and their
feature encoders (`tools/encode_t3.py`, `tools/encode_sitr.py`,
`tools/live_features.py`; HTT = the in-graph `tf9dpe` convention).

## Short answer

**T3 and SITR consume RAW tactile images β€” no reference / background-frame
subtraction.** Each follows its authors' published input convention. Only the
HTT features use reference subtraction.

## Operation chains (from `/images/raw_<f>`: 240Γ—320 uint8 grayscale)

### T3 (matches the paper's T3 probe rows exactly)

1. uint8 β†’ float, Γ·255 β†’ [0,1]
2. grayscale replicated to 3 channels
3. bilinear resize β†’ 224Γ—224
4. normalize `(x βˆ’ 0.49611) / 0.18685` β€” **per-dataset stats computed once on
   the screwing corpus** (T3's convention is per-dataset stats on raw RGB;
   borrowing another dataset's stats was measured to put inputs several Οƒ
   off-distribution during the tactile_fusion probe work). These constants are
   training-fixed β€” never recompute at deployment.
5. β†’ T3-medium encoder (domain `mini`) β†’ 9-block trunk β†’ **non-affine**
   LayerNorm β†’ cls token β†’ 768-d per finger (Γ—5 fingers, order 0..4 β†’ 3840-d).

### SITR

1–3. same Γ·255 β†’ 3-channel replicate β†’ 224Γ—224
4. normalize with SITR's training stats applied on the [0,1] tensor:
   mean = (βˆ’1.2223, βˆ’1.8114, βˆ’1.7090), std = (11.7932, 12.7956, 13.6452)
   (from `gsrl/dataloaders.py`).
5. β†’ `SITR_base(num_calibration=0).forward_encoder(x, c=None)`, cls token β†’
   768-d per finger. **`num_calibration=0`**: SITR's sensor-invariance
   mechanism normally consumes calibration frames (its own flavor of reference
   frames; the tactile_fusion probe baselines used 18) β€” the ACT feature
   encoding runs it calibration-free.

### HTT (`tf9dpe`, for contrast β€” the only one with reference frames)

1. per-episode **background frame subtraction** (the `tactile_bg_224` group;
   at deployment: captured at the settled episode-start pose)
2. division by image std
3. **2-frame temporal stacking** (`tactile_image_pair`)
4. resize 224Γ—224 β†’ 9dtact ViT encoder + shared trunk β†’ 192-d per finger
   (Γ—5 β†’ 960-d).

## Why the asymmetry is deliberate

Each encoder receives its authors' released input convention β€” the defensible
way to run baselines. Imposing HTT's bg-subtraction on T3/SITR would deviate
from their published usage and invite a "you ran them wrong" objection.

## Practical implication for robot deployment

T3/SITR policies have **no reference-frame mechanism at all**: if the
fingertip cameras' resting appearance drifts between the training data and
the rig (lighting, gel wear, mounting), their features shift with nothing to
anchor them. Interpret on-robot behavior differences with this in mind. The
episode-start background capture fix (see `real_deploy_script/DEPLOY_FIXES.md`)
matters only for HTT β€” T3/SITR need no bg capture, by design.