erensckin/hand2throw-lam

Hand2Throw: latent action models

Small latent action models (~9 M parameters each) from the Hand2Throw project, in the style of LAPO (Schmidt & Jiang, Learning to Act without Actions, ICLR 2024): inverse and forward dynamics trained on video alone, with a quantised latent bottleneck. Code and full write-up: https://github.com/erensckin/hand2throw

Each model learns, without action labels, to describe the change between two video frames (t and t+4, 96×96) as a small discrete latent action, and to predict frame t+4 from frame t and that latent:

1. Action encoder (inverse dynamics): both frames stacked → 4-level CNN → 2-layer MLP → 4 tokens × 3 numbers.
2. FSQ quantiser: 5 levels per number → 125 codes per token (~28 bits per frame pair).
3. Context encoder: frame t → multi-scale features.
4. Decoder (forward dynamics, the world model): U-Net, conditioned on the latent at every scale (FiLM); predicts the change image added to frame t.

Checkpoints

All trained 15k steps, k = 4 (0.2 s at 20 Hz).

Folder	Trained on
robot_side	The robot's side camera (sim)
human_cam1_masked, human_cam2_masked	Operator webcam / phone video with everything outside the tracked hand blacked out
human_cam1_silhouette, human_cam2_silhouette	Control: the same hand region as a plain white shape, no hand pixels

On held-out episodes, linear probes from the masked webcam + phone latents recover the operator's commanded motion about as well as the MediaPipe hand landmarks (R² x / z / gripper 0.69 / 0.67 / 0.59 vs 0.69 / 0.67 / 0.87). The silhouette control reaches 0.68 / 0.69 / 0.55, so the signal comes mainly from the tracker-placed hand region. Latents from unmasked video reach only 0.1–0.2.

Run it (robot camera)

The robot-camera model can be reproduced end to end from the published data:

git clone https://github.com/erensckin/hand2throw && cd hand2throw && uv sync --locked
uv run hf download erensckin/hand2throw-teleop-demos --repo-type dataset --local-dir data/throw_ketchup
uv run hf download erensckin/hand2throw-lam --include "robot_side/*" --local-dir outputs/lam
uv run python scripts/lam.py extract --raw data/throw_ketchup/extra    # 96 px caches of the robot cameras (~2.7 GB)
uv run python scripts/lam_predict.py --sources robot_side --fidelity   # imagined vs real frames: PSNR, plots, videos
uv run python scripts/lam.py probe --sources robot_side                 # latent -> robot action R²

The human-video models need the operator recordings, which are available on request. The silhouette models can be probed without them, using hand2throw-hand-silhouettes:

uv run hf download erensckin/hand2throw-hand-silhouettes --repo-type dataset --local-dir data/lam_cache
uv run hf download erensckin/hand2throw-lam --include "human_cam*_silhouette/*" --local-dir outputs/lam
uv run python scripts/lam.py probe --sources human_cam1_silhouette,human_cam2_silhouette

Loading

import torch
from lam_model import LatentActionModel  # included in this repo

ckpt = torch.load("robot_side/lam.pt", map_location="cpu", weights_only=True)
model = LatentActionModel(**ckpt["cfg"])
model.load_state_dict(ckpt["model"])

Each lam.pt holds {"model": state_dict, "cfg", "k", "source", "step"}. No training data is stored in it. Frame preprocessing, probes and world-model rollouts are in scripts/lam.py and scripts/lam_predict.py in the GitHub repo.