yrpark/openvitac-act-gelsight

ACT baselines for OpenViTac (GelSight, simulation)

ACT (Action Chunking with Transformers) policies trained on two simulation tasks from the OpenViTac benchmark, using the released FVL-Lab/OpenViTac dataset (Simulation/GelSight_mini). These are reproduction runs, not the authors' checkpoints.

Task (OpenViTac env)	Dataset folder	Demos	Best val loss	Sim success (policy_last.ckpt)
can_empty_select	empty_can_selection	100	0.0706 (epoch 64)	12/42 (28.6%)
hardness_classify	hardness_classify	100	0.0982 (epoch 114)	50/50 (100%)

Training of the other nine released tasks was started but stopped before completion, so only these two are included.

Files

<task>/policy_best.ckpt     lowest validation loss
<task>/policy_last.ckpt     after 20,000 steps (the checkpoint that was evaluated)
<task>/dataset_stats.pkl    qpos/action normalization stats (required at inference)
<task>/train_val_*.png      training curves
train_config.yml            ACT config used for both tasks (OpenViTac policy/ACT/train_config.yml)
convert_lerobot.py          LeRobot v2.1 -> ACT HDF5 converter used to build the training data
eval/                       per-episode evaluation logs and summary

To use with OpenViTac (commit 5220aaf), place a task folder at policy/ACT/act_ckpt/act-<task>/gelsight-100/train_config/ and run EP_NUM=100 TRAIN_CONFIG=train_config python scripts/eval_policy.py <task> gelsight ACT/deploy. The env must use task_config/gelsight.yml (skip_pre_move: true) and policy/task_settings.json needs an entry for the task ("camera_type": "all").

Training setup

• Model: OpenViTac's ACT (train_config.yml): ResNet-18 backbones for two cameras (head, wrist) and two tactile images (left/right GelSight rgb_marker), chunk size 50, temporal aggregation, 20,000 steps, batch 64, lr 1e-5, seed 0. One RTX 3090, about 2.5 h per task.
• No pretrained tactile encoder. train_config.yml points to encoder/checkpoints/resnet18/.../best.pth, which is not released; the code silently skips a missing file, so the tactile ResNet-18 starts from its default initialization.
• Data conversion (convert_lerobot.py): the released dataset has no action column and stores images as MP4. State/qpos = 7 arm joints + left finger position in metres (matches the env's embodiment/joint[:8]), action = next frame's state (60 Hz, i.e. every 2 sim steps), images RGB at native resolution. The dataset includes the motion-planned pre-move segment and has no phase column, so all frames are used; evaluation starts from the same initial pose because skip_pre_move is on.

Evaluation

50 episodes per task (seeds 1000000-1000049, 5 chunks of 10), success as defined by each OpenViTac task's check_success. Episodes that raise an environment exception are excluded by OpenViTac's eval loop.

• can_empty_select: 8 of 50 episodes hit the env's 120 s scene-reset timeout (an infrastructure failure, not a policy decision) and are excluded, leaving 42. Those runs shared the GPU with training, which slowed resets. The policy grasps, carries and drops a can reliably; failures were wrong-can choices.
• hardness_classify: evaluated with the reset limit raised to 300 s; no exclusions. Hard/soft labels were mixed across seeds (e.g. 7/3 and 6/4 in the first two chunks), and success requires the block on the correct plate, so 100% reflects correct touch-based classification.

Per-episode lines are in eval/<task>_c<k>.txt.