RammusLeo/TouchAnnotator

Multi-Source Tactile Supervision
with Privileged-View Distillation

Turn a single egocentric video into bimanual tactile annotations.

💻 GitHub Code  ·  🤗 Checkpoint  ·  🚀 Quick Start  ·  🧠 Method  ·  🎨 Results  ·  📖 Inference Guide  ·  中文

_{Concept illustration from the paper (Fig. 1). Heatmaps are schematic, not model predictions or sensor measurements.}

✨ From seeing interaction to estimating touch

TouchAnnotator predicts a 21 × 21 relative-pressure field and contact-probability field for each hand. A single RGB video supplies both the visual input and WiLoR-estimated hand poses. The model uses compact past-clip memory to add temporal context.

🤲 Bimanual fields	🎥 Ego-only deployment	🧩 Multi-source learning
Separate left/right pressure and contact predictions.	One video in; no wrist cameras or tactile sensors at inference.	Six training datasets spanning gloves, bare hands and dexterous robots.

_{Predicting touch from the ego view. EgoTouch qualitative comparisons from Fig. 3; column labels retain the paper's input-view protocols.}

🤗 Released checkpoint

This repository provides the TouchAnnotator Full student, with a Qwen3.5-4B visual-language backbone and tactile prediction heads. It annotates a single person's hands in an egocentric video using this pipeline:

Your RGB video ──→ WiLoR hand poses ──→ TouchAnnotator ──→ Pressure + contact
                        └──────── same RGB video ────────┘

Item	Release details
Checkpoint	model.pth, 9.40 GB (9,401,275,903 bytes)
Contents	Student model_state_dict; original tensor dtypes; optimizer and EMA teacher excluded
Backbone	Qwen3.5-4B
Inputs	Ego RGB video and WiLoR-estimated 3D hand poses
Outputs	Per-hand 21 × 21 relative pressure and contact probability
Temporal context	Eight-frame clips; two preceding clips, 16 memory tokens each
Matching files	config.yaml · manifest.json
Version	v1.0; public downloads, no login required

Use the bundled inference code and demo.py. This is a custom PyTorch checkpoint; it is not directly loadable through AutoModel.from_pretrained() or a Transformers text-generation pipeline. The source bundle excludes model binaries; the downloader fetches TouchAnnotator, Qwen and WiLoR separately.

Checkpoint provenance and SHA256

Exported from epoch 2 of the Full-student run used by the research code's Full/FPHA evaluation scripts:

20260824_003600_qwen3.5_4b_tactile_prediction_qwen35_full_causal_memory_opd_full_qwen

The export contains 1,218 tensors and preserves their values and dtypes. Load it with the matching configuration; the demo checks keys and shapes strictly.

SHA256(model.pth)
16ee5a3779e1e21e6b1102375dad2e8455c28cfbb90c12c7f2d310e91bee8170

python scripts/download_assets.py verifies the downloaded size and SHA256 against manifest.json.

🤗 Optional 0813 qualitative checkpoint

This repository remains the 0824 Full student. An earlier 0813 checkpoint is separately available at RammusLeo/TouchAnnotator-0813: 951 MB, Qwen vision tower + temporal/query decoder, eight-frame overlapping windows, no Full-Qwen language reasoner, clip memory or OPD. Historical selected examples showed stronger visible pressure signals; this is not a claim of higher accuracy.

0813 outputs pressure and pressure-thresholded contact state (default 0.1), without an independent contact-probability field. Install the environment and WiLoR as below, use source version v1.1.0 or later, then:

python scripts/download_assets.py --variant 0813
python scripts/check_assets.py --variant 0813
CUDA_VISIBLE_DEVICES=0 python demo.py --variant 0813 \
  --video /path/to/bimanual_video.mp4 --output-dir outputs/my_video_0813

0813 inference guide · Source v1.1.0. Omit --variant to keep using this repository's default 0824 checkpoint.

🚀 Quick start

Linux · NVIDIA CUDA GPU · Python 3.10. Run the commands below from the extracted source bundle root (TouchAnnotator/). The tested runtime uses PyTorch 2.7.1, torchvision 0.22.1 and Transformers 5.3.0, with SDPA/BF16 inference. See validation notes for hardware and installation coverage.

0 · Get the inference code

Clone the public inference repository and open its root:

git clone https://github.com/RammusLeo/TouchAnnotator.git
cd TouchAnnotator

You can also download the source ZIP, extract it, and open TouchAnnotator/. The source bundle mirrors the GitHub release and excludes model binaries.

1 · Set up the environment

conda create -n touchannotator python=3.10 -y
conda activate touchannotator
python -m pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r requirements.txt
conda install -c conda-forge ffmpeg -y
mkdir -p third_party
git clone https://github.com/rolpotamias/WiLoR.git third_party/WiLoR
git -C third_party/WiLoR checkout fcb911312a38fa8badd30d9656a167485d61b8f9

WiLoR is imported directly from its pinned checkout.

2 · Get the model assets

python scripts/download_assets.py
python scripts/check_assets.py

For an optional download mirror in mainland China:

HF_ENDPOINT=https://hf-mirror.com python scripts/download_assets.py

The downloader uses anonymous requests (token=False). HF-Mirror usage.

This single command downloads the TouchAnnotator Full student, its matching configuration and manifest, the official Qwen3.5-4B backbone, and WiLoR weights. It verifies the student checkpoint's SHA256. All downloads are public and require no Hugging Face login.

🤗 Model repository · Checkpoint · 9.40 GB · Config · SHA256 manifest

Download only TouchAnnotator when upstream models are already available

python scripts/download_assets.py --only-model

Equivalent Hugging Face CLI command:

hf download RammusLeo/TouchAnnotator model.pth config.yaml manifest.json --revision v1.0 --local-dir checkpoints/touchannotator

The downloader pins the model release to v1.0, Qwen to 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, and WiLoR to 99fe3d7acff8104ecca1055df7467709506c2fa6.

📁 Expected model directory layout

checkpoints/
  touchannotator/
    model.pth                 # 9,401,275,903 bytes, student only
    config.yaml               # exact architecture + preprocessing
    manifest.json             # release SHA256
  Qwen3.5-4B/                 # official HF snapshot, including tokenizer and weights
third_party/WiLoR/
  wilor/
  pretrained_models/
    model_config.yaml         # from the pinned WiLoR checkout
    wilor_final.ckpt
    detector.pt
  mano_data/
    mano_mean_params.npz

Reproducible release: model.pth, config.yaml and manifest.json are versioned together in release v1.0. Keep this set together.

The adapter can reconstruct MANO from buffers contained in the official WiLoR checkpoint, matching the existing evaluation adapter. If using a separately obtained MANO model, place MANO_RIGHT.pkl under mano_data/ or pass --mano-dir. Upstream model terms still apply; see third-party notices.

Use the Full student checkpoint with its matching configuration.

3 · Annotate your video

Use a first-person RGB video showing one person's left and right hands. Start with a short, clear clip with limited occlusion.

CUDA_VISIBLE_DEVICES=0 python demo.py \
  --video /path/to/bimanual_video.mp4 \
  --output-dir outputs/my_video

Start with a short smoke test:

CUDA_VISIBLE_DEVICES=0 python demo.py \
  --video /path/to/bimanual_video.mp4 \
  --max-frames 50 --output-dir outputs/smoke

--max-frames counts source frames, before sampling. Use --no-video to skip MP4 rendering. Output directories containing predictions are protected from accidental overwrite. WiLoR and TouchAnnotator run sequentially to release pose-estimator GPU memory before tactile inference.

🧠 How it works

Training and deployment overview, reproduced from Fig. 2.

1. Unify tactile supervision. A canonical bimanual field brings together measured glove pressure, geometry-derived contact and robot fingertip deformation, with source-aware validity masks.
2. Learn from privileged views. An EMA teacher observes ego and wrist video during training; the deployed student uses ego RGB and estimated pose.
3. Carry temporal context. Two previous clips are compressed into 16 tokens each. Separate heads predict relative pressure and contact probability.

The current eight-frame clip uses bidirectional reasoning. Memory is past-only across clips; this is an offline video annotator, not frame-online inference.

🎨 Qualitative results

Bare-hand contact on FPHA

Fig. 3, lower panels. Bare-hand contact estimates compared with geometry-derived reference labels.

Across hands and embodiments

Fig. 5. Two sets of manipulation examples span dexterous robot hands, bare hands and tactile gloves. Colors are sensor-relative and do not establish a shared physical pressure scale.

Comparison with measured touch

Fig. 4. Synchronized video and glove recordings with example predictions and measured maps. The collection contains 897 retained sequences across 30 tasks; the paper's shared quantitative comparison uses an 860-sequence subset.

All images above are extracted from the author-provided manuscript. Figure provenance and asset details.

📦 What you get

Output	Contents
prediction.mp4	Original video beside left/right pressure maps
predictions.npz	Pressure, contact probability, valid-cell masks, hand poses and timing
wilor_poses.jsonl	WiLoR joints, per-hand detection validity and gap-filling flags
metadata.json	Frame counts, pose coverage, precision and peak GPU memory

import numpy as np
prediction = np.load('outputs/my_video/predictions.npz')
pressure = prediction['pressure']               # (N, 2, 21, 21)
contact = prediction['contact_probability']     # (N, 2, 21, 21)
valid_cells = prediction['valid_cells']          # (2, 21, 21)

Hand order is left, right. Pressure is normalized and sensor-relative, not force in Newtons or pressure in kPa. Check pose validity for occluded or missed hands. Full output schema, preprocessing and limitations →

🔬 Release validation

10 regression tests passed · 3 real-video smoke tests completed. The bimanual test detected left/right hands in 21/25 and 20/25 sampled frames. Peak allocated GPU memory was approximately 10.26 GB on an H200. These checks validate the inference path and output contracts; they do not reproduce every paper table or establish a minimum VRAM requirement.

The release contains the Full student, its matching inference config and a SHA256 manifest. Export preserves tensor values and dtypes while removing optimizer and teacher state. Detailed validation record →

⚠️ Intended use and limitations

• Intended for research and offline tactile annotation of egocentric manipulation videos. The current eight-frame clip and pose smoothing can use later frames within the video.
• Pressure is a normalized, sensor-relative prediction, with no calibrated physical unit. Contact probabilities are not calibrated guarantees of contact.
• Occlusion, gloves, fast motion, off-screen hands and multiple people can reduce pose coverage and prediction quality. Use the saved detection and pose-validity flags when interpreting results.
• The demo retains sampled frames in host RAM. Split long recordings into shorter videos; each run resets temporal memory.
• The release smoke tests verify the executable inference path. Reproducing paper metrics requires the corresponding datasets, splits and evaluation protocols.

🤝 Acknowledgements

Built on the TouchAnything research codebase, WiLoR, and Qwen3.5-4B. The training formulation draws on EgoTouch, HumanTouch, HOI4D, TACO, OakInk2 and T-Rex.

Original project code retains the MIT license. The student contains Qwen-derived parameters; the upstream Qwen license notice is retained. WiLoR and MANO retain their respective terms. The code license does not relicense model weights or third-party assets; see third-party notices.