You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

V8 on the lab Franka FR3: deployment bundle (code + weights)

This file is self-contained. It is both AGENTS.md (for a coding agent such as Codex / Claude Code running on the robot PC) and README.md (for humans). Hugging Face repo: lin-v8-robot/v8-real-fr3 (private; needs a read token with access to the lin-v8-robot organization). No cloud or cluster access is needed; everything below runs on the robot PC.

0. Purpose and what to use first

V8 = a two-pass action policy (AP) + a 4D world model (WM). Per control decision: (1) the AP's first pass proposes a 16-step action chunk (the "action history"); (2) the WM predicts the 3D motion of scene query points under that chunk; (3) a small cross-attention readout lets the AP's second pass refine the chunk from that predicted motion. The first pass is a frozen copy of the base policy ap2, so the second pass can only refine, never lose the base behaviour.

Use first: final_pertask/ (final configuration, 2026-11-05): one model per task, two heads (choose one per run):

  • final_pertask/dp/<task>.pt = U-Net diffusion-policy head (DP)
  • final_pertask/dit/<task>.pt = DiT head
  • shared: wm/wm_Rq.pt (real-robot world model), deploy_assets_robotiq.npz (Robotiq 2F-85 gripper meshes, D455 camera, per-task validity maps), calibration/ (D455 extrinsics / intrinsics of the recordings)
  • code: final_pertask/code_v8.tar.gz (full repository: v8/ package, scripts/ incl. scripts/deploy_real/, configs/)

Tasks (<task>) and the exact instruction (--language) each model was trained with:

order task --language (exact) notes
1 close_drawer close the bottom drawer of the cabinet (or close the top drawer of the cabinet) push only, no grasp; easiest
2 pick_place pick up the cup and place it on the plate single grasp
3 cup_insertion put the brown cup into the blue mug precise insertion
4 drawer put the cup in the bottom drawer of the cabinet and close it longest: grasp, place, push

Place the objects as in setup/<task>_setup.jpg (first frame, middle, end, wrist view of a recorded demonstration).

1. File map

final_pertask/dp/{close_drawer,cup_insertion,drawer,pick_place}.pt    DP head, final config (USE THESE)
final_pertask/dit/{close_drawer,cup_insertion,drawer,pick_place}.pt   DiT head, final config (USE THESE)
final_pertask/code_v8.tar.gz                                          code for final_pertask (both heads)
MANIFEST_final_pertask.txt                                            sha256 + bytes of the final_pertask files
wm/wm_Rq.pt                                                           world model (Robotiq labels) used by every two-pass model
deploy_assets_robotiq.npz                                             gripper meshes + camera + validity maps (use with wm_Rq)
calibration/d455_extrinsics.yaml, d455_aligned_depth_intrinsics.yaml  camera calibration of the recordings
setup/<task>_setup.jpg                                                object placement photos
--- older sets (keep for comparison only, never mix files across sets) ---
final/relm_a.pt, final/ap2.pt, final/relm_b.pt, final/graspft.pt       11-04 multi-task policies (one model for all 4 tasks); code/v8_code.tar.gz
ap_rq/<head>_<task>.pt                                                 10-31 per-task Robotiq set (unet/dit); code/v8_code.tar.gz
ap/<head>_<task>.pt, wm/wm_Bp.pt, deploy_assets.npz                    oldest Franka-hand-label set (--gripper franka)

Inside code_v8.tar.gz: scripts/deploy_real/serve_v8_ws.py (WebSocket policy server, the only process that runs a model), scripts/deploy_real/parity_check.py (offline replay / checks), scripts/deploy_real/ws_client_v8.yaml (config for the lab ROS 2 bridge robot_ws_client), v8/policy/* (AP), v8/wm/* (WM), configs/*.yaml. Code version: final_pertask/code_v8.tar.gz (449 files, sha256 in MANIFEST_final_pertask.txt): the cloud working tree v8-main-1105dit of branch v8 (based on commit 63de88b, DiT consequence cross-attention adapters). The DiT checkpoints were trained with exactly this tree; the DP checkpoints with an earlier tree of the same branch; all 8 load and replay with this code (section 3).

2. Environment (once, about 20 min)

Tested with Python 3.10, CUDA 12.8, torch 2.8.0 (+cu128). Versions used in replay: numpy 2.2.6, scipy 1.15.3, h5py 3.16, pyyaml 6.0, opencv (headless) 5.0, timm 1.0.30, transformers 5.19, diffusers 0.41, sentencepiece 0.2.2, protobuf 7.36, open3d 0.20, einops 0.8.2, websockets 16.1, huggingface_hub 1.33.

mkdir -p ~/ros_ml_data/checkpoints && cd ~/ros_ml_data/checkpoints
pip install -U "huggingface_hub[cli]"; hf auth login                         # read token with access to org lin-v8-robot
hf download lin-v8-robot/v8-real-fr3 --local-dir ~/ros_ml_data/checkpoints/v8 \
    --include "final_pertask/*" "wm/wm_Rq.pt" "deploy_assets_robotiq.npz" "calibration/*" "setup/*" "MANIFEST_final_pertask.txt" "*.md"
cd ~/ros_ml_data/checkpoints/v8 && sha256sum -c <(awk '{print $1"  "$3}' MANIFEST_final_pertask.txt)   # all must print OK
mkdir -p ~/v8code && tar xzf ~/ros_ml_data/checkpoints/v8/final_pertask/code_v8.tar.gz -C ~/v8code && cd ~/v8code
conda create -n v8deploy python=3.10 -y && conda activate v8deploy
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install numpy scipy h5py pyyaml opencv-python-headless timm transformers diffusers sentencepiece protobuf open3d einops websockets
hf download google/siglip-base-patch16-224; hf download timm/resnet50.a1_in1k   # tokenizer + backbone definitions (public)

After this the server runs offline (HF_HUB_OFFLINE=1).

3. Offline check BEFORE touching the robot (5 minutes)

  1. Server starts and loads the model (GPU): run the section-4 server command; it must print [v8] ... mode=two_pass ... and [v8] listening on ws://....

  2. Optional replay of recorded episodes (only if the recorded LeRobot data is on this PC): python scripts/deploy_real/parity_check.py replay --uri ws://127.0.0.1:8765 --task <task> --ap-ckpt <ckpt> --data <v8 episode root> --raw <LeRobot root> --stream --out replay.json Expected (our offline replay of the validation episodes, server stream mode, two-pass):

    task (validation episode) dp: pos cm / rot deg / gripper agree dit: pos / rot / gripper 11-04 relm_a, same episode
    close_drawer (demo_1, 36 decisions) 1.43 / 1.97 / 1.000 1.39 / 1.82 / 1.000 1.39 / 1.76 / 1.000
    pick_place (demo_21, 36) 1.24 / 1.57 / 0.989 1.13 / 1.57 / 0.989 1.06 / 1.75 / 0.967
    cup_insertion (demo_17, 62) 1.19 / 1.99 / 0.977 1.15 / 2.36 / 0.990 1.25 / 2.33 / 0.977
    drawer (demo_18, 69) 1.66 / 2.22 / 0.971 1.78 / 2.22 / 0.977 1.70 / 2.14 / 0.983
    all 4 (203 decisions) 1.40 / 1.99 / 0.981 1.40 / 2.08 / 0.987 relm_a, all validation: 1.39 / 2.08 / 0.983

    pos / rot = mean error of the predicted targets vs the recorded demonstration; about 1-2 cm and 1.5-2.5 deg is normal. Several cm or gripper agreement below 0.9 means a wrong file set, wrong --language, or wrong images.

4. Run on the robot (three terminals)

# A: robot + cameras (lab procedure): Franka Desk unlocked, then
robot_up
ros2 launch realsense2_camera rs_launch.py serial_no:="'254922307708'" camera_namespace:=d455 camera_name:=camera align_depth.enable:=true
ros2 launch realsense2_camera rs_launch.py serial_no:="'260322276493'" camera_namespace:=d405 camera_name:=camera align_depth.enable:=true

# B: V8 server (GPU). Pick HEAD=dp or dit and the task row of section 0.
conda activate v8deploy; cd ~/v8code; C=~/ros_ml_data/checkpoints/v8
HEAD=dp; TASK=close_drawer; LANG="close the bottom drawer of the cabinet"
HF_HUB_OFFLINE=1 python scripts/deploy_real/serve_v8_ws.py --ap-ckpt $C/final_pertask/$HEAD/$TASK.pt --wm-ckpt $C/wm/wm_Rq.pt \
    --assets $C/deploy_assets_robotiq.npz --task $TASK --language "$LANG" --device cuda:0 --port 8765
#   other rows: TASK=pick_place LANG="pick up the cup and place it on the plate"
#               TASK=cup_insertion LANG="put the brown cup into the blue mug"
#               TASK=drawer LANG="put the cup in the bottom drawer of the cabinet and close it"
#   latency test: add --timing-log ~/v8_timing.jsonl (per-request ms printed on disconnect)

# C: lab bridge, DRY RUN first (gripper_type = the mounted gripper: Robotiq 2F-85 -> robotiq)
source ~/ros_ml_ws/install/setup.bash
ros2 launch robot_ws_client ws_client.launch.py config:=$HOME/v8code/scripts/deploy_real/ws_client_v8.yaml \
    server_uri:=ws://127.0.0.1:8765 dry_run:=true gripper_type:=robotiq camera_profile:=d455_d405
ros2 topic echo /robot_ws_client/debug_target_pose      # targets near the gripper, smooth, ~10 Hz

Then, with the workspace clear and one person at the emergency stop, restart terminal C with dry_run:=false. Only one node may publish /cartesian_impedance_node/target_pose (stop teleop / replay first).

Latency (offline replay, per decision, two-pass = draft + WM + refine): on our shared, fully loaded cloud GPU the median server time per decision was 0.8-1.6 s (DiT head) and 7-12 s (DP head); these numbers are inflated by other jobs on the same GPU. The earlier per-task DP models ran well under 0.5 s per decision on an unshared GPU. Measure on the robot PC with --timing-log. GPU memory: about 3-5 GB per server. If it is too slow on the robot PC, run the first pass only (no WM): replace the model by the old final/ap2.pt with --mode draft and no --wm-ckpt/--assets (this is the base policy without the world model).

5. Safety

  • Start in dry_run:=true, check targets in RViz / debug_target_pose, then run with the e-stop in hand; start with the objects placed exactly as in setup/<task>_setup.jpg.
  • The D455 must not have moved since recording (the policies only saw that view); D455 third view + D405 wrist, both 848x480 rgb8, 10 Hz.
  • Gripper: Robotiq 2F-85 (recorded opening 0..0.04 per finger = 850 steps of the 85 mm stroke). Wire convention: 0 = close, 1 = open.
  • Stop immediately if the arm leaves the table area, moves fast toward the cabinet, or the gripper closes on nothing repeatedly.

6. Troubleshooting

  • Empty actions in the reply = observation rejected (wrong image size, missing gripper width): both camera topics must publish 848x480 rgb8, gripper_type must match the mounted gripper.
  • Robot moves the wrong way or jumps: back to dry run, compare debug_target_pose with /cartesian_impedance_node/current_pose (quaternions x, y, z, w).
  • --language must be exactly the string of section 0 (the model was trained with these instructions).
  • Model load error about missing keys: the checkpoint and the code must come from the same set (final_pertask/*.pt with final_pertask/code_v8.tar.gz; older sets with code/v8_code.tar.gz).

7. Do NOT change

  • Do not mix files across sets (final_pertask / final / ap_rq / ap), do not use wm/wm_Bp.pt or deploy_assets.npz with these models.
  • Do not change image sizes, camera order, control rate (10 Hz), --history, --exec-steps, the action gains or the gripper convention.
  • Do not edit v8/ or configs/; the checkpoints store their own config and the code reads it.

8. Record

Per trial: task, head (dp / dit), mode (two-pass / draft), success yes / no, time to success, failure type. About 20 trials per task.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support