Instructions to use mfincato/leap_t4a_groot_n17_40k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use mfincato/leap_t4a_groot_n17_40k with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
GR00T N1.7 · LEAP hand · touch-for-all tactile state · step 40 000
Fine-tune of nvidia/GR00T-N1.7-3B through LeRobot's groot
policy, on the LEAP-hand dataset HOUJC128/leap_jc_one_clean (101 episodes, 10 fps, task
"Grasp the object and put it in the plate"). The three fingertip tactile sensors enter the model as state,
not as images: each stream is encoded with the touch-for-all v26 e32 cross-sensor encoder and reduced with
one PCA shared by the three fingers.
This repo: step 40 000 — end of training. The held-out eval loss rose from 0.095 (step 4 000) to 0.21: compare it on the robot with the 10k checkpoint.
| checkpoint | held-out eval loss | note |
|---|---|---|
mfincato/leap_t4a_groot_n17_10k |
0.105 | best saved (minimum 0.095 at step 4 000, not saved) |
mfincato/leap_t4a_groot_n17_40k |
0.210 | end of training, overfitted on eval loss |
Eval loss per 1 000 steps: validation_history.json. Training curves:
https://wandb.ai/hsp-lerobot/leap_groot/runs/leap_t4a_groot_40k_b32-69
Inputs and outputs
| key | shape | content |
|---|---|---|
observation.images.left |
(480, 640, 3) uint8 RGB | scene camera (letterboxed to 256 × 256 inside the preprocessor) |
observation.state |
(125,) float32 | 0–6 Panda joints, 7–22 LEAP joints leap.joint.0..15, 23–28 EE pose ee.x ee.y ee.z ee.wx ee.wy ee.wz, 29–124 tactile: thumb pc00–31, index pc00–31, middle pc00–31 |
task |
str | "Grasp the object and put it in the plate" |
action |
(22,) per step | ee.x ee.y ee.z ee.wx ee.wy ee.wz, leap.action.0..15 (absolute, same units as the dataset) |
The policy predicts a chunk of 40 and executes the first 5 (n_action_steps=5), at 10 Hz.
Normalisation of state and action (q01/q99 → [-1, 1], clipped) is inside the saved pre/post-processors.
Tactile state (values 29–124)
| finger | sensor | source in the recording |
|---|---|---|
| thumb | GelSight Mini | RGB 320 × 240 portrait (stored as AV1 video) |
| index | Daimon (serial M2505150136) | deformation field (240, 320, 2) int16 µm, 30 Hz (tactile_raw/) |
| middle | DIGIT | RGB 320 × 240 portrait (stored as AV1 video) |
Per finger and per control step:
- touch-for-all encoder (repo
gabrielecaddeo/touch-for-all, branchvalidation-program, commit0f9b5e6, weightsweights/touch4all_v26_encoder.pt; trunk = AnyTouch 2 CLIP-B/16, gated repoxxuan01/AnyTouch2-Model): window of 8 frames 0.1 s apart, reference = a no-contact frame taken at the start of every episode (training used the first frame of each episode) → 256-d embedding; (embedding − mean) @ components.Twithtactile/tactile_pca.npz(one mean / basis for all fingers, K = 32) → 32 values, concatenated thumb | index | middle.
tactile/deploy_tactile.py does exactly this (TactileState). It needs the encoder environment
(transformers 4.42, numpy < 2), which conflicts with the GR00T environment (transformers 5.x): run it in a
separate process and send the 96 values to the policy process.
tactile/encode_lerobot_tactile.py and tactile/prepare_t4a.py are the scripts that produced the training
data (reference for offline checks); tactile/tactile_pca_report.json has the variance kept (pooled 93.5 %).
LeRobot version
Trained with the fork LeoGori/lerobot, branch feat/ds_no_haptics, commit 8a845fa
(lerobot_patch/source_revision.txt), plus two patches to apply in this order from the repo root:
git checkout 8a845fad173b9f6087ba89136f41d566458bc545
git apply lerobot_patch/source_existing_changes.patch # adds GrootConfig.exclude_state (unused here, False)
cd src && patch -p1 < ../lerobot_patch/source.patch # 256x256 letterbox per view; drop_n_last_frames=0
The letterbox patch matters at inference: without it the image is resized differently from training.
Inference (policy process)
import torch
from huggingface_hub import snapshot_download
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.groot.modeling_groot import GrootPolicy
repo = "mfincato/leap_t4a_groot_n17_40k"
policy = GrootPolicy.from_pretrained(repo).to("cuda").eval() # downloads nvidia/GR00T-N1.7-3B the first time
pre, post = make_pre_post_processors(policy.config, pretrained_path=repo)
policy.reset() # at the start of every episode (also reset the tactile references)
obs = {
"observation.images.left": img, # torch float (1, 3, 480, 640) in [0, 1], RGB
"observation.state": torch.cat([proprio_29, tactile_96]).unsqueeze(0), # (1, 125) float32
"task": ["Grasp the object and put it in the plate"],
}
with torch.no_grad():
action = post(policy.select_action(pre(obs))) # (1, 22); one action per call, chunk of 5 queued
Tactile process (encoder environment, PYTHONPATH with transformers 4.42 / numpy < 2,
TOUCH4ALL=<clone of touch-for-all @ 0f9b5e6>):
from deploy_tactile import TactileState # tactile/ folder of this repo (reads tactile_pca.npz next to it)
ts = TactileState(device="cuda")
ts.set_references({"thumb": gelsight_rest, "index": daimon_rest, "middle": digit_rest}) # nothing touching
# every sensor frame: ts.push("thumb", frame, t_seconds) (same for "index", "middle")
# every control step: tactile_96 = ts.state_values() (needs >= 0.1 s of frames after the reset)
check_inference.py was used to verify, on CPU, that this repo (with base_model_path = nvidia/GR00T-N1.7-3B) predicts exactly the same actions as the original local checkpoint (max difference
0.0 on held-out frames).
Training
40 000 steps, batch 32 (1 × A100 80 GB, no accumulation), AdamW lr 1e-4 cosine, 2 000 warm-up steps,
weight decay 1e-5, bf16 with fp32 parameters, seed 42; frozen LLM and vision tower, trained projector,
action head (diffusion) and VL layer norm; 90 train / 11 held-out episodes
([3, 13, 14, 17, 28, 31, 35, 69, 81, 86, 94]). Images: only the left camera (the tactile videos are not
given as images). Full config: train_config.json.
Known limits
- The DIGIT on the middle finger is not the unit the encoder was trained on (resting Fz offset ≈ 0.7 N, likely over-reads). The Daimon is a validated unit.
- GelSight and DIGIT were encoded from AV1 video (GOP 2, frame-alternating noise); at run time the frames come uncompressed from the sensors, so their embeddings will be slightly cleaner than in training.
- Eval loss is a flow-matching loss on held-out episodes, not a success rate.
- Downloads last month
- 8
Model tree for mfincato/leap_t4a_groot_n17_40k
Base model
nvidia/GR00T-N1.7-3B