GR00T N1.7 · LEAP hand · touch-for-all tactile state · step 40 000

Fine-tune of nvidia/GR00T-N1.7-3B through LeRobot's groot policy, on the LEAP-hand dataset HOUJC128/leap_jc_one_clean (101 episodes, 10 fps, task "Grasp the object and put it in the plate"). The three fingertip tactile sensors enter the model as state, not as images: each stream is encoded with the touch-for-all v26 e32 cross-sensor encoder and reduced with one PCA shared by the three fingers.

This repo: step 40 000 — end of training. The held-out eval loss rose from 0.095 (step 4 000) to 0.21: compare it on the robot with the 10k checkpoint.

checkpoint held-out eval loss note
mfincato/leap_t4a_groot_n17_10k 0.105 best saved (minimum 0.095 at step 4 000, not saved)
mfincato/leap_t4a_groot_n17_40k 0.210 end of training, overfitted on eval loss

Eval loss per 1 000 steps: validation_history.json. Training curves: https://wandb.ai/hsp-lerobot/leap_groot/runs/leap_t4a_groot_40k_b32-69

Inputs and outputs

key shape content
observation.images.left (480, 640, 3) uint8 RGB scene camera (letterboxed to 256 × 256 inside the preprocessor)
observation.state (125,) float32 0–6 Panda joints, 7–22 LEAP joints leap.joint.0..15, 23–28 EE pose ee.x ee.y ee.z ee.wx ee.wy ee.wz, 29–124 tactile: thumb pc00–31, index pc00–31, middle pc00–31
task str "Grasp the object and put it in the plate"
action (22,) per step ee.x ee.y ee.z ee.wx ee.wy ee.wz, leap.action.0..15 (absolute, same units as the dataset)

The policy predicts a chunk of 40 and executes the first 5 (n_action_steps=5), at 10 Hz. Normalisation of state and action (q01/q99 → [-1, 1], clipped) is inside the saved pre/post-processors.

Tactile state (values 29–124)

finger sensor source in the recording
thumb GelSight Mini RGB 320 × 240 portrait (stored as AV1 video)
index Daimon (serial M2505150136) deformation field (240, 320, 2) int16 µm, 30 Hz (tactile_raw/)
middle DIGIT RGB 320 × 240 portrait (stored as AV1 video)

Per finger and per control step:

  1. touch-for-all encoder (repo gabrielecaddeo/touch-for-all, branch validation-program, commit 0f9b5e6, weights weights/touch4all_v26_encoder.pt; trunk = AnyTouch 2 CLIP-B/16, gated repo xxuan01/AnyTouch2-Model): window of 8 frames 0.1 s apart, reference = a no-contact frame taken at the start of every episode (training used the first frame of each episode) → 256-d embedding;
  2. (embedding − mean) @ components.T with tactile/tactile_pca.npz (one mean / basis for all fingers, K = 32) → 32 values, concatenated thumb | index | middle.

tactile/deploy_tactile.py does exactly this (TactileState). It needs the encoder environment (transformers 4.42, numpy < 2), which conflicts with the GR00T environment (transformers 5.x): run it in a separate process and send the 96 values to the policy process.

tactile/encode_lerobot_tactile.py and tactile/prepare_t4a.py are the scripts that produced the training data (reference for offline checks); tactile/tactile_pca_report.json has the variance kept (pooled 93.5 %).

LeRobot version

Trained with the fork LeoGori/lerobot, branch feat/ds_no_haptics, commit 8a845fa (lerobot_patch/source_revision.txt), plus two patches to apply in this order from the repo root:

git checkout 8a845fad173b9f6087ba89136f41d566458bc545
git apply lerobot_patch/source_existing_changes.patch        # adds GrootConfig.exclude_state (unused here, False)
cd src && patch -p1 < ../lerobot_patch/source.patch           # 256x256 letterbox per view; drop_n_last_frames=0

The letterbox patch matters at inference: without it the image is resized differently from training.

Inference (policy process)

import torch
from huggingface_hub import snapshot_download
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.groot.modeling_groot import GrootPolicy

repo = "mfincato/leap_t4a_groot_n17_40k"
policy = GrootPolicy.from_pretrained(repo).to("cuda").eval()   # downloads nvidia/GR00T-N1.7-3B the first time
pre, post = make_pre_post_processors(policy.config, pretrained_path=repo)

policy.reset()                      # at the start of every episode (also reset the tactile references)
obs = {
    "observation.images.left": img,             # torch float (1, 3, 480, 640) in [0, 1], RGB
    "observation.state": torch.cat([proprio_29, tactile_96]).unsqueeze(0),   # (1, 125) float32
    "task": ["Grasp the object and put it in the plate"],
}
with torch.no_grad():
    action = post(policy.select_action(pre(obs)))   # (1, 22); one action per call, chunk of 5 queued

Tactile process (encoder environment, PYTHONPATH with transformers 4.42 / numpy < 2, TOUCH4ALL=<clone of touch-for-all @ 0f9b5e6>):

from deploy_tactile import TactileState          # tactile/ folder of this repo (reads tactile_pca.npz next to it)
ts = TactileState(device="cuda")
ts.set_references({"thumb": gelsight_rest, "index": daimon_rest, "middle": digit_rest})   # nothing touching
# every sensor frame:      ts.push("thumb", frame, t_seconds)  (same for "index", "middle")
# every control step:      tactile_96 = ts.state_values()      (needs >= 0.1 s of frames after the reset)

check_inference.py was used to verify, on CPU, that this repo (with base_model_path = nvidia/GR00T-N1.7-3B) predicts exactly the same actions as the original local checkpoint (max difference 0.0 on held-out frames).

Training

40 000 steps, batch 32 (1 × A100 80 GB, no accumulation), AdamW lr 1e-4 cosine, 2 000 warm-up steps, weight decay 1e-5, bf16 with fp32 parameters, seed 42; frozen LLM and vision tower, trained projector, action head (diffusion) and VL layer norm; 90 train / 11 held-out episodes ([3, 13, 14, 17, 28, 31, 35, 69, 81, 86, 94]). Images: only the left camera (the tactile videos are not given as images). Full config: train_config.json.

Known limits

  • The DIGIT on the middle finger is not the unit the encoder was trained on (resting Fz offset ≈ 0.7 N, likely over-reads). The Daimon is a validated unit.
  • GelSight and DIGIT were encoded from AV1 video (GOP 2, frame-alternating noise); at run time the frames come uncompressed from the sensors, so their embeddings will be slightly cleaner than in training.
  • Eval loss is a flow-matching loss on held-out episodes, not a success rate.
Downloads last month
8
Safetensors
Model size
3B params
Tensor type
F32
·
Video Preview
loading

Model tree for mfincato/leap_t4a_groot_n17_40k

Finetuned
(330)
this model