YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
V8 on the lab Franka FR3: deployment bundle (code + weights)
This file is self-contained. It is both AGENTS.md (for a coding agent such as Codex / Claude Code running on the robot PC) and
README.md (for humans). Hugging Face repo: lin-v8-robot/v8-real-fr3 (private; needs a read token with access to the
lin-v8-robot organization). No cloud or cluster access is needed; everything below runs on the robot PC.
0. Purpose and what to use first
V8 = a two-pass action policy (AP) + a 4D world model (WM). Per control decision: (1) the AP's first pass proposes a 16-step action
chunk (the "action history"); (2) the WM predicts the 3D motion of scene query points under that chunk; (3) a small cross-attention
readout lets the AP's second pass refine the chunk from that predicted motion. The first pass is a frozen copy of the base policy
ap2, so the second pass can only refine, never lose the base behaviour.
Use first: final_pertask/ (final configuration, 2026-11-05): one model per task, two heads (choose one per run):
final_pertask/dp/<task>.pt= U-Net diffusion-policy head (DP)final_pertask/dit/<task>.pt= DiT head- shared:
wm/wm_Rq.pt(real-robot world model),deploy_assets_robotiq.npz(Robotiq 2F-85 gripper meshes, D455 camera, per-task validity maps),calibration/(D455 extrinsics / intrinsics of the recordings) - code:
final_pertask/code_v8.tar.gz(full repository:v8/package,scripts/incl.scripts/deploy_real/,configs/)
Tasks (<task>) and the exact instruction (--language) each model was trained with:
| order | task | --language (exact) |
notes |
|---|---|---|---|
| 1 | close_drawer | close the bottom drawer of the cabinet (or close the top drawer of the cabinet) |
push only, no grasp; easiest |
| 2 | pick_place | pick up the cup and place it on the plate |
single grasp |
| 3 | cup_insertion | put the brown cup into the blue mug |
precise insertion |
| 4 | drawer | put the cup in the bottom drawer of the cabinet and close it |
longest: grasp, place, push |
Place the objects as in setup/<task>_setup.jpg (first frame, middle, end, wrist view of a recorded demonstration).
1. File map
final_pertask/dp/{close_drawer,cup_insertion,drawer,pick_place}.pt DP head, final config (USE THESE)
final_pertask/dit/{close_drawer,cup_insertion,drawer,pick_place}.pt DiT head, final config (USE THESE)
final_pertask/code_v8.tar.gz code for final_pertask (both heads)
MANIFEST_final_pertask.txt sha256 + bytes of the final_pertask files
wm/wm_Rq.pt world model (Robotiq labels) used by every two-pass model
deploy_assets_robotiq.npz gripper meshes + camera + validity maps (use with wm_Rq)
calibration/d455_extrinsics.yaml, d455_aligned_depth_intrinsics.yaml camera calibration of the recordings
setup/<task>_setup.jpg object placement photos
--- older sets (keep for comparison only, never mix files across sets) ---
final/relm_a.pt, final/ap2.pt, final/relm_b.pt, final/graspft.pt 11-04 multi-task policies (one model for all 4 tasks); code/v8_code.tar.gz
ap_rq/<head>_<task>.pt 10-31 per-task Robotiq set (unet/dit); code/v8_code.tar.gz
ap/<head>_<task>.pt, wm/wm_Bp.pt, deploy_assets.npz oldest Franka-hand-label set (--gripper franka)
Inside code_v8.tar.gz: scripts/deploy_real/serve_v8_ws.py (WebSocket policy server, the only process that runs a model),
scripts/deploy_real/parity_check.py (offline replay / checks), scripts/deploy_real/ws_client_v8.yaml (config for the lab ROS 2
bridge robot_ws_client), v8/policy/* (AP), v8/wm/* (WM), configs/*.yaml.
Code version: final_pertask/code_v8.tar.gz (449 files, sha256 in MANIFEST_final_pertask.txt): the cloud working tree v8-main-1105dit of branch v8 (based on commit 63de88b, DiT consequence cross-attention adapters). The DiT checkpoints were trained with exactly this tree; the DP checkpoints with an earlier tree of the same branch; all 8 load and replay with this code (section 3).
2. Environment (once, about 20 min)
Tested with Python 3.10, CUDA 12.8, torch 2.8.0 (+cu128). Versions used in replay: numpy 2.2.6, scipy 1.15.3, h5py 3.16, pyyaml 6.0, opencv (headless) 5.0, timm 1.0.30, transformers 5.19, diffusers 0.41, sentencepiece 0.2.2, protobuf 7.36, open3d 0.20, einops 0.8.2, websockets 16.1, huggingface_hub 1.33.
mkdir -p ~/ros_ml_data/checkpoints && cd ~/ros_ml_data/checkpoints
pip install -U "huggingface_hub[cli]"; hf auth login # read token with access to org lin-v8-robot
hf download lin-v8-robot/v8-real-fr3 --local-dir ~/ros_ml_data/checkpoints/v8 \
--include "final_pertask/*" "wm/wm_Rq.pt" "deploy_assets_robotiq.npz" "calibration/*" "setup/*" "MANIFEST_final_pertask.txt" "*.md"
cd ~/ros_ml_data/checkpoints/v8 && sha256sum -c <(awk '{print $1" "$3}' MANIFEST_final_pertask.txt) # all must print OK
mkdir -p ~/v8code && tar xzf ~/ros_ml_data/checkpoints/v8/final_pertask/code_v8.tar.gz -C ~/v8code && cd ~/v8code
conda create -n v8deploy python=3.10 -y && conda activate v8deploy
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install numpy scipy h5py pyyaml opencv-python-headless timm transformers diffusers sentencepiece protobuf open3d einops websockets
hf download google/siglip-base-patch16-224; hf download timm/resnet50.a1_in1k # tokenizer + backbone definitions (public)
After this the server runs offline (HF_HUB_OFFLINE=1).
3. Offline check BEFORE touching the robot (5 minutes)
Server starts and loads the model (GPU): run the section-4 server command; it must print
[v8] ... mode=two_pass ...and[v8] listening on ws://....Optional replay of recorded episodes (only if the recorded LeRobot data is on this PC):
python scripts/deploy_real/parity_check.py replay --uri ws://127.0.0.1:8765 --task <task> --ap-ckpt <ckpt> --data <v8 episode root> --raw <LeRobot root> --stream --out replay.jsonExpected (our offline replay of the validation episodes, server stream mode, two-pass):task (validation episode) dp: pos cm / rot deg / gripper agree dit: pos / rot / gripper 11-04 relm_a, same episode close_drawer (demo_1, 36 decisions) 1.43 / 1.97 / 1.000 1.39 / 1.82 / 1.000 1.39 / 1.76 / 1.000 pick_place (demo_21, 36) 1.24 / 1.57 / 0.989 1.13 / 1.57 / 0.989 1.06 / 1.75 / 0.967 cup_insertion (demo_17, 62) 1.19 / 1.99 / 0.977 1.15 / 2.36 / 0.990 1.25 / 2.33 / 0.977 drawer (demo_18, 69) 1.66 / 2.22 / 0.971 1.78 / 2.22 / 0.977 1.70 / 2.14 / 0.983 all 4 (203 decisions) 1.40 / 1.99 / 0.981 1.40 / 2.08 / 0.987 relm_a, all validation: 1.39 / 2.08 / 0.983 pos / rot = mean error of the predicted targets vs the recorded demonstration; about 1-2 cm and 1.5-2.5 deg is normal. Several cm or gripper agreement below 0.9 means a wrong file set, wrong
--language, or wrong images.
4. Run on the robot (three terminals)
# A: robot + cameras (lab procedure): Franka Desk unlocked, then
robot_up
ros2 launch realsense2_camera rs_launch.py serial_no:="'254922307708'" camera_namespace:=d455 camera_name:=camera align_depth.enable:=true
ros2 launch realsense2_camera rs_launch.py serial_no:="'260322276493'" camera_namespace:=d405 camera_name:=camera align_depth.enable:=true
# B: V8 server (GPU). Pick HEAD=dp or dit and the task row of section 0.
conda activate v8deploy; cd ~/v8code; C=~/ros_ml_data/checkpoints/v8
HEAD=dp; TASK=close_drawer; LANG="close the bottom drawer of the cabinet"
HF_HUB_OFFLINE=1 python scripts/deploy_real/serve_v8_ws.py --ap-ckpt $C/final_pertask/$HEAD/$TASK.pt --wm-ckpt $C/wm/wm_Rq.pt \
--assets $C/deploy_assets_robotiq.npz --task $TASK --language "$LANG" --device cuda:0 --port 8765
# other rows: TASK=pick_place LANG="pick up the cup and place it on the plate"
# TASK=cup_insertion LANG="put the brown cup into the blue mug"
# TASK=drawer LANG="put the cup in the bottom drawer of the cabinet and close it"
# latency test: add --timing-log ~/v8_timing.jsonl (per-request ms printed on disconnect)
# C: lab bridge, DRY RUN first (gripper_type = the mounted gripper: Robotiq 2F-85 -> robotiq)
source ~/ros_ml_ws/install/setup.bash
ros2 launch robot_ws_client ws_client.launch.py config:=$HOME/v8code/scripts/deploy_real/ws_client_v8.yaml \
server_uri:=ws://127.0.0.1:8765 dry_run:=true gripper_type:=robotiq camera_profile:=d455_d405
ros2 topic echo /robot_ws_client/debug_target_pose # targets near the gripper, smooth, ~10 Hz
Then, with the workspace clear and one person at the emergency stop, restart terminal C with dry_run:=false.
Only one node may publish /cartesian_impedance_node/target_pose (stop teleop / replay first).
Latency (offline replay, per decision, two-pass = draft + WM + refine): on our shared, fully loaded cloud GPU the median server time per decision was 0.8-1.6 s (DiT head) and 7-12 s (DP head); these numbers are inflated by other jobs on the same GPU. The earlier per-task DP models ran well under 0.5 s per decision on an unshared GPU. Measure on the robot PC with --timing-log. GPU memory: about 3-5 GB per server.
If it is too slow on the robot PC, run the first pass only (no WM): replace the model by the old final/ap2.pt with --mode draft
and no --wm-ckpt/--assets (this is the base policy without the world model).
5. Safety
- Start in
dry_run:=true, check targets in RViz /debug_target_pose, then run with the e-stop in hand; start with the objects placed exactly as insetup/<task>_setup.jpg. - The D455 must not have moved since recording (the policies only saw that view); D455 third view + D405 wrist, both 848x480 rgb8, 10 Hz.
- Gripper: Robotiq 2F-85 (recorded opening 0..0.04 per finger = 850 steps of the 85 mm stroke). Wire convention: 0 = close, 1 = open.
- Stop immediately if the arm leaves the table area, moves fast toward the cabinet, or the gripper closes on nothing repeatedly.
6. Troubleshooting
- Empty
actionsin the reply = observation rejected (wrong image size, missing gripper width): both camera topics must publish 848x480 rgb8,gripper_typemust match the mounted gripper. - Robot moves the wrong way or jumps: back to dry run, compare
debug_target_posewith/cartesian_impedance_node/current_pose(quaternions x, y, z, w). --languagemust be exactly the string of section 0 (the model was trained with these instructions).- Model load error about missing keys: the checkpoint and the code must come from the same set (
final_pertask/*.ptwithfinal_pertask/code_v8.tar.gz; older sets withcode/v8_code.tar.gz).
7. Do NOT change
- Do not mix files across sets (final_pertask / final / ap_rq / ap), do not use
wm/wm_Bp.ptordeploy_assets.npzwith these models. - Do not change image sizes, camera order, control rate (10 Hz),
--history,--exec-steps, the action gains or the gripper convention. - Do not edit
v8/orconfigs/; the checkpoints store their own config and the code reads it.
8. Record
Per trial: task, head (dp / dit), mode (two-pass / draft), success yes / no, time to success, failure type. About 20 trials per task.