Add Xtrainer2 real robot source and Chinese handoff guide

README.md

@@ -25,6 +25,7 @@ This package contains the server-authoritative Florence–Bridge OPTR code, its
 
 ## What is included
 
+- [`real_robot/`](real_robot/README.md)：Xtrainer2 真机数据转换、训练、双臂动作适配和 policy server 源码，以及中文交接说明。
 - `common_start_20k/checkpoints/steps_20000_accelerate/`: **full resumable state** (model, optimizer, scheduler metadata and per-rank RNG state). Start every new continuation arm here.
 - `checkpoints/uniform/steps_20500_pytorch_model.pt`: model-only Uniform +500 snapshot.
 - `checkpoints/residual/steps_20500_pytorch_model.pt`: model-only Residual-OPTR +500 snapshot.

real_robot/AGENTS.md

@@ -0,0 +1,10 @@
+# Xtrainer2 真机源码交接
+
+本目录用于真机数据处理、训练与 policy server 部署。先阅读 `README.md` 和 `SOURCE_FILES.json`，从 `source/` 进入原始源码根目录。
+
+- 使用本目录内的 `starVLA`、`retime` 和 `deployment` 包。
+- 按所选真机配置核对 GPU 数、global batch、训练预算、数据混合和恢复 step。
+- 模型权重、`config.yaml` 和 `dataset_statistics.json` 成套部署。
+- 相机顺序为 high、left wrist、right wrist，机器人侧 16 维 xyzw 布局，模型内部 20 维 rotation-6D。
+- 硬件驱动和执行循环接入实际机械臂客户端，执行动作前核对位置单位、夹爪约定、左右臂顺序和控制频率。
+- 源码及来源文件保留上游内容，新增实验使用独立输出目录。

real_robot/README.md

@@ -0,0 +1,72 @@
+# Xtrainer2 真机代码交接
+
+这份源码来自你的 GitHub 仓库，整理在 HF 的独立目录 `real_robot/`。训练与推理服务以 `ReTime_source/main` 的 Xtrainer2 实现为主体，原开发仓库 `OPTR-StarVLA/main` 提供 HDF5 数据转换与 LeRobot 格式验证工具。逐文件来源见 [SOURCE_FILES.json](SOURCE_FILES.json)。
+
+## 代码入口
+
+| 用途 | 文件 |
+|---|---|
+| HDF5 → LeRobot 数据转换 | `source/scripts/convert_xtrainer2_hdf5_to_lerobot.py` |
+| 数据格式与四元数约定验证 | `source/scripts/validate_xtrainer2_lerobot.py` |
+| 数据注册、三相机及双臂状态配置 | `source/examples/Xtrainer2/train_files/data_registry/data_config.py` |
+| 数据 support 构建 | `source/examples/Xtrainer2/train_files/build_pack_phone_support.py` 和 `build_pick_bottle_support.py` |
+| 训练启动 | `source/examples/Xtrainer2/train_files/run_xtrainer2_train.sh` |
+| ReTime proposal 与 continuation | `source/retime/cli.py` 和 `source/scripts/run_xtrainer2_arm.sh` |
+| 真机 policy server 启动 | `source/examples/Xtrainer2/train_files/run_xtrainer2_policy_server.sh` |
+| 双臂动作与四元数转换 | `source/deployment/model_server/xtrainer2_command.py` |
+| WebSocket 客户端通信工具 | `source/deployment/model_server/tools/websocket_policy_client.py` |
+
+`source/` 保留上游包布局。进入该目录再运行命令，使 `starVLA`、`retime` 和 `deployment` 使用这份源码。
+
+## 环境与依赖
+
+目标运行环境为 Linux、Python ≥ 3.10。原依赖列表为 [requirements.txt](source/requirements.txt) 和 [requirements-xtrainer2.txt](source/requirements-xtrainer2.txt)，其中 PyTorch 为 2.7.1，Transformers 为 4.57.0，Accelerate 为 1.13.0。
+
+双臂四元数与 rotation-6D 转换需要匹配 Python、PyTorch、CUDA 的 PyTorch3D。数据转换另需 `h5py`、`pandas`、Pillow 和可执行的 `ffmpeg`。按目标 GPU 环境安装相应依赖。
+
+## 数据与训练配置
+
+训练配置放在 `source/configs/`，包含 pack-phone、pick-bottle、smoke 和从 pack-phone common start 延续的配置。原配置使用 Florence、8 GPU、每卡 micro-batch 4、累积 4，global batch 为 128。pack-phone 的完整训练预算为 52,380 updates，warmup 为 5,000。continuation 配置从 `steps_17460_accelerate` 延续，训练 seed 为 117465，基础训练 seed 为 42。
+
+运行前按配置设置 `FLORENCE_MODEL_PATH`、`XTRAINER2_DATA_ROOT`、`OPTR_RUN_ROOT` 及数据 support 路径。训练入口会检查配置实际引用的环境变量。具体实现见 [Xtrainer2 数据说明](source/examples/Xtrainer2/README.md)。
+
+## 推理服务
+
+准备真机 checkpoint 对应的运行目录，模型、配置、归一化统计保持成套。
+
+```text
+your_real_robot_run/
+├── config.yaml
+├── dataset_statistics.json
+└── checkpoints/
+    └── your_model.pt
+```
+
+在下载后的 `real_robot/source` 中执行以下命令，将示例路径改为实际路径。
+
+```bash
+export OPTR_PYTHON=/path/to/your/python
+export FLORENCE_MODEL_PATH=/path/to/Florence-2-large
+export TMPDIR="$PWD/runtime_tmp"
+mkdir -p "$TMPDIR"
+
+bash examples/Xtrainer2/train_files/run_xtrainer2_policy_server.sh \
+  /path/to/your_real_robot_run/checkpoints/your_model.pt 0 10093
+```
+
+服务使用 BF16，并在连接握手中返回动作维数、相机顺序、四元数约定和 action chunk 长度。
+
+## 机械臂客户端接口
+
+相机顺序为 `cam_high`、`cam_left_wrist`、`cam_right_wrist`，图像使用 HWC RGB uint8。客户端提供 `image`、`lang` 和原始 16 维 `state`。双臂状态与动作按每臂位置 3 维、四元数 4 维、夹爪 1 维排列，机器人侧四元数顺序为 `xyzw`。服务端负责归一化和坐标表示转换，模型内部使用 20 维 rotation-6D 表示，返回动作使用机器人侧的 16 维布局。
+
+pick-bottle 的相对末端动作由服务端结合当前状态还原为绝对动作。真实机械臂驱动、相机采集和执行循环由独立硬件客户端完成，需接入师兄实际使用的控制程序。上游接口说明见 [ROBOT.md](source/examples/Xtrainer2/ROBOT.md)。
+
+## 本次交接验证
+
+- 129 个源文件与来源清单的字节数一致，逐文件来源见 `SOURCE_FILES.json`。
+- 95 个 Python 文件通过语法检查，7 个 YAML、1 个 JSON 和 1 个 TOML 文件通过解析检查。
+- 8 个 shell 入口通过 `bash -n`，`python -m retime --help` 可正常运行。
+- 使用 jlu66 现有 Python 环境在 CPU 上执行 13 项配置测试，全部通过，覆盖双臂维数、global batch、梯度累积、ZeRO-3 权重保存设置、训练种子和 continuation 参数。
+
+实际模型加载、GPU 推理和机械臂动作执行需在配套设备与 checkpoint 上验证。

real_robot/SOURCE_FILES.json

@@ -0,0 +1,909 @@
+{
+  "schema_version": 1,
+  "description": "Xtrainer2 real-robot training and policy-serving source handoff",
+  "files": [
+    {
+      "path": "source/configs/deepspeed_florence_8gpu.yaml",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "configs/deepspeed_florence_8gpu.yaml",
+      "bytes": 702
+    },
+    {
+      "path": "source/configs/florence_xtrainer2_pack_phone_residual_20ep_to_60ep.yaml",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "configs/florence_xtrainer2_pack_phone_residual_20ep_to_60ep.yaml",
+      "bytes": 2837
+    },
+    {
+      "path": "source/configs/florence_xtrainer2_pack_phone_residual_resume_smoke.yaml",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "configs/florence_xtrainer2_pack_phone_residual_resume_smoke.yaml",
+      "bytes": 2967
+    },
+    {
+      "path": "source/configs/florence_xtrainer2_pack_phone_smoke.yaml",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "configs/florence_xtrainer2_pack_phone_smoke.yaml",
+      "bytes": 3954
+    },
+    {
+      "path": "source/configs/florence_xtrainer2_pack_phone_uniform.yaml",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "configs/florence_xtrainer2_pack_phone_uniform.yaml",
+      "bytes": 3898
+    },
+    {
+      "path": "source/configs/florence_xtrainer2_pick_bottle_rel_aug_smoke.yaml",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "configs/florence_xtrainer2_pick_bottle_rel_aug_smoke.yaml",
+      "bytes": 3703
+    },
+    {
+      "path": "source/configs/florence_xtrainer2_pick_bottle_rel_aug_uniform.yaml",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "configs/florence_xtrainer2_pick_bottle_rel_aug_uniform.yaml",
+      "bytes": 4104
+    },
+    {
+      "path": "source/deployment/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/__init__.py",
+      "bytes": 0
+    },
+    {
+      "path": "source/deployment/model_server/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/__init__.py",
+      "bytes": 0
+    },
+    {
+      "path": "source/deployment/model_server/policy_norm_processor.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/policy_norm_processor.py",
+      "bytes": 22282
+    },
+    {
+      "path": "source/deployment/model_server/policy_wrapper.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/policy_wrapper.py",
+      "bytes": 12974
+    },
+    {
+      "path": "source/deployment/model_server/server_policy.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/server_policy.py",
+      "bytes": 2741
+    },
+    {
+      "path": "source/deployment/model_server/tools/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/tools/__init__.py",
+      "bytes": 0
+    },
+    {
+      "path": "source/deployment/model_server/tools/image_tools.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/tools/image_tools.py",
+      "bytes": 4765
+    },
+    {
+      "path": "source/deployment/model_server/tools/msgpack_numpy.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/tools/msgpack_numpy.py",
+      "bytes": 2016
+    },
+    {
+      "path": "source/deployment/model_server/tools/websocket_policy_client.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/tools/websocket_policy_client.py",
+      "bytes": 2728
+    },
+    {
+      "path": "source/deployment/model_server/tools/websocket_policy_server.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/tools/websocket_policy_server.py",
+      "bytes": 5813
+    },
+    {
+      "path": "source/deployment/model_server/xtrainer2_command.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "deployment/model_server/xtrainer2_command.py",
+      "bytes": 3595
+    },
+    {
+      "path": "source/examples/Xtrainer2/INFERENCE.md",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/INFERENCE.md",
+      "bytes": 180
+    },
+    {
+      "path": "source/examples/Xtrainer2/README.md",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/README.md",
+      "bytes": 1511
+    },
+    {
+      "path": "source/examples/Xtrainer2/ROBOT.md",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/ROBOT.md",
+      "bytes": 1124
+    },
+    {
+      "path": "source/examples/Xtrainer2/train_files/build_pack_phone_support.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/train_files/build_pack_phone_support.py",
+      "bytes": 3018
+    },
+    {
+      "path": "source/examples/Xtrainer2/train_files/build_pick_bottle_support.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/train_files/build_pick_bottle_support.py",
+      "bytes": 3071
+    },
+    {
+      "path": "source/examples/Xtrainer2/train_files/data_registry/data_config.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/train_files/data_registry/data_config.py",
+      "bytes": 6348
+    },
+    {
+      "path": "source/examples/Xtrainer2/train_files/modality.json",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/train_files/modality.json",
+      "bytes": 1877
+    },
+    {
+      "path": "source/examples/Xtrainer2/train_files/run_xtrainer2_policy_server.sh",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/train_files/run_xtrainer2_policy_server.sh",
+      "bytes": 4398
+    },
+    {
+      "path": "source/examples/Xtrainer2/train_files/run_xtrainer2_train.sh",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "examples/Xtrainer2/train_files/run_xtrainer2_train.sh",
+      "bytes": 3016
+    },
+    {
+      "path": "source/LICENSE",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "LICENSE",
+      "bytes": 1387
+    },
+    {
+      "path": "source/licenses/Diffusion-Policy-LICENSE",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "licenses/Diffusion-Policy-LICENSE",
+      "bytes": 1106
+    },
+    {
+      "path": "source/licenses/Isaac-GR00T-LICENSE",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "licenses/Isaac-GR00T-LICENSE",
+      "bytes": 10808
+    },
+    {
+      "path": "source/licenses/MODIFIED_FILES.txt",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "licenses/MODIFIED_FILES.txt",
+      "bytes": 765
+    },
+    {
+      "path": "source/licenses/msgpack-numpy-LICENSE",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "licenses/msgpack-numpy-LICENSE",
+      "bytes": 1551
+    },
+    {
+      "path": "source/licenses/OpenVLA-LICENSE",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "licenses/OpenVLA-LICENSE",
+      "bytes": 1105
+    },
+    {
+      "path": "source/licenses/X-VLA-LICENSE",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "licenses/X-VLA-LICENSE",
+      "bytes": 11357
+    },
+    {
+      "path": "source/NOTICE",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "NOTICE",
+      "bytes": 2060
+    },
+    {
+      "path": "source/pyproject.toml",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "pyproject.toml",
+      "bytes": 1790
+    },
+    {
+      "path": "source/README.md",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "README.md",
+      "bytes": 10922
+    },
+    {
+      "path": "source/requirements-qwen.txt",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "requirements-qwen.txt",
+      "bytes": 269
+    },
+    {
+      "path": "source/requirements-xtrainer2.txt",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "requirements-xtrainer2.txt",
+      "bytes": 440
+    },
+    {
+      "path": "source/requirements.txt",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "requirements.txt",
+      "bytes": 574
+    },
+    {
+      "path": "source/retime/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/__init__.py",
+      "bytes": 160
+    },
+    {
+      "path": "source/retime/__main__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/__main__.py",
+      "bytes": 79
+    },
+    {
+      "path": "source/retime/checkpoint.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/checkpoint.py",
+      "bytes": 864
+    },
+    {
+      "path": "source/retime/cli.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/cli.py",
+      "bytes": 13905
+    },
+    {
+      "path": "source/retime/commands/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/commands/__init__.py",
+      "bytes": 43
+    },
+    {
+      "path": "source/retime/commands/proposal.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/commands/proposal.py",
+      "bytes": 25822
+    },
+    {
+      "path": "source/retime/commands/shuffle.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/commands/shuffle.py",
+      "bytes": 1482
+    },
+    {
+      "path": "source/retime/objective.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/objective.py",
+      "bytes": 6506
+    },
+    {
+      "path": "source/retime/prepare.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/prepare.py",
+      "bytes": 1145
+    },
+    {
+      "path": "source/retime/pretrain.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/pretrain.py",
+      "bytes": 7183
+    },
+    {
+      "path": "source/retime/proposal.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/proposal.py",
+      "bytes": 9294
+    },
+    {
+      "path": "source/retime/sampler.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/sampler.py",
+      "bytes": 13229
+    },
+    {
+      "path": "source/retime/scoring/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/__init__.py",
+      "bytes": 218
+    },
+    {
+      "path": "source/retime/scoring/cheap_proposals.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/cheap_proposals.py",
+      "bytes": 7068
+    },
+    {
+      "path": "source/retime/scoring/cnf_likelihood.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/cnf_likelihood.py",
+      "bytes": 10056
+    },
+    {
+      "path": "source/retime/scoring/discrete_flow_likelihood.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/discrete_flow_likelihood.py",
+      "bytes": 7055
+    },
+    {
+      "path": "source/retime/scoring/flow_residual.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/flow_residual.py",
+      "bytes": 9357
+    },
+    {
+      "path": "source/retime/scoring/partial_gradient.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/partial_gradient.py",
+      "bytes": 4067
+    },
+    {
+      "path": "source/retime/scoring/residual_cache.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/residual_cache.py",
+      "bytes": 2815
+    },
+    {
+      "path": "source/retime/scoring/shuffle.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/shuffle.py",
+      "bytes": 4229
+    },
+    {
+      "path": "source/retime/scoring/starvla_likelihood.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/starvla_likelihood.py",
+      "bytes": 5643
+    },
+    {
+      "path": "source/retime/scoring/utility_audit.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "retime/scoring/utility_audit.py",
+      "bytes": 18264
+    },
+    {
+      "path": "source/scripts/build_residual_temporal_proposal.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "scripts/build_residual_temporal_proposal.py",
+      "bytes": 243
+    },
+    {
+      "path": "source/scripts/build_shuffle_temporal_proposal.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "scripts/build_shuffle_temporal_proposal.py",
+      "bytes": 252
+    },
+    {
+      "path": "source/scripts/convert_xtrainer2_hdf5_to_lerobot.py",
+      "repository": "miaochengy-pngu/OPTR-StarVLA",
+      "ref": "main",
+      "source_path": "scripts/convert_xtrainer2_hdf5_to_lerobot.py",
+      "bytes": 26880
+    },
+    {
+      "path": "source/scripts/launch_pack_phone_optr_proposal.sh",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "scripts/launch_pack_phone_optr_proposal.sh",
+      "bytes": 247
+    },
+    {
+      "path": "source/scripts/launch_pack_phone_residual_20ep_to_60ep.sh",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "scripts/launch_pack_phone_residual_20ep_to_60ep.sh",
+      "bytes": 2187
+    },
+    {
+      "path": "source/scripts/launch_pack_phone_uniform.sh",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "scripts/launch_pack_phone_uniform.sh",
+      "bytes": 1885
+    },
+    {
+      "path": "source/scripts/launch_pick_bottle_rel_aug_smoke.sh",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "scripts/launch_pick_bottle_rel_aug_smoke.sh",
+      "bytes": 1156
+    },
+    {
+      "path": "source/scripts/launch_pick_bottle_rel_aug_uniform.sh",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "scripts/launch_pick_bottle_rel_aug_uniform.sh",
+      "bytes": 1441
+    },
+    {
+      "path": "source/scripts/run_xtrainer2_arm.sh",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "scripts/run_xtrainer2_arm.sh",
+      "bytes": 6276
+    },
+    {
+      "path": "source/scripts/validate_xtrainer2_lerobot.py",
+      "repository": "miaochengy-pngu/OPTR-StarVLA",
+      "ref": "main",
+      "source_path": "scripts/validate_xtrainer2_lerobot.py",
+      "bytes": 5871
+    },
+    {
+      "path": "source/starVLA/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/__init__.py",
+      "bytes": 0
+    },
+    {
+      "path": "source/starVLA/dataloader/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/__init__.py",
+      "bytes": 4672
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/__init__.py",
+      "bytes": 0
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/data_config.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/data_config.py",
+      "bytes": 35501
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/datasets.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/datasets.py",
+      "bytes": 144422
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/embodiment_tags.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/embodiment_tags.py",
+      "bytes": 2459
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/mixtures.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/mixtures.py",
+      "bytes": 19387
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/registry.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/registry.py",
+      "bytes": 5960
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/schema.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/schema.py",
+      "bytes": 8590
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/transform/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/transform/__init__.py",
+      "bytes": 1261
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/transform/base.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/transform/base.py",
+      "bytes": 5154
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/transform/concat.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/transform/concat.py",
+      "bytes": 9371
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/transform/state_action.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/transform/state_action.py",
+      "bytes": 27803
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/transform/video.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/transform/video.py",
+      "bytes": 24512
+    },
+    {
+      "path": "source/starVLA/dataloader/gr00t_lerobot/video.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/gr00t_lerobot/video.py",
+      "bytes": 14043
+    },
+    {
+      "path": "source/starVLA/dataloader/infoflow_sampler.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/infoflow_sampler.py",
+      "bytes": 180
+    },
+    {
+      "path": "source/starVLA/dataloader/lerobot_datasets.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/lerobot_datasets.py",
+      "bytes": 6050
+    },
+    {
+      "path": "source/starVLA/dataloader/relative_pose.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/relative_pose.py",
+      "bytes": 3359
+    },
+    {
+      "path": "source/starVLA/dataloader/temporal_proposal.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/dataloader/temporal_proposal.py",
+      "bytes": 181
+    },
+    {
+      "path": "source/starVLA/model/framework/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/framework/__init__.py",
+      "bytes": 202
+    },
+    {
+      "path": "source/starVLA/model/framework/base_framework.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/framework/base_framework.py",
+      "bytes": 11351
+    },
+    {
+      "path": "source/starVLA/model/framework/share_tools.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/framework/share_tools.py",
+      "bytes": 21898
+    },
+    {
+      "path": "source/starVLA/model/framework/VLM4A/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/framework/VLM4A/__init__.py",
+      "bytes": 296
+    },
+    {
+      "path": "source/starVLA/model/framework/VLM4A/QwenGR00T.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/framework/VLM4A/QwenGR00T.py",
+      "bytes": 15342
+    },
+    {
+      "path": "source/starVLA/model/modules/action_model/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/modules/action_model/__init__.py",
+      "bytes": 122
+    },
+    {
+      "path": "source/starVLA/model/modules/action_model/flow_matching_head/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/modules/action_model/flow_matching_head/__init__.py",
+      "bytes": 679
+    },
+    {
+      "path": "source/starVLA/model/modules/action_model/flow_matching_head/action_encoder.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/modules/action_model/flow_matching_head/action_encoder.py",
+      "bytes": 3431
+    },
+    {
+      "path": "source/starVLA/model/modules/action_model/flow_matching_head/cross_attention_dit.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/modules/action_model/flow_matching_head/cross_attention_dit.py",
+      "bytes": 14773
+    },
+    {
+      "path": "source/starVLA/model/modules/action_model/GR00T_ActionHeader.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/modules/action_model/GR00T_ActionHeader.py",
+      "bytes": 21024
+    },
+    {
+      "path": "source/starVLA/model/modules/vlm/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/modules/vlm/__init__.py",
+      "bytes": 481
+    },
+    {
+      "path": "source/starVLA/model/modules/vlm/Florence2.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/modules/vlm/Florence2.py",
+      "bytes": 12888
+    },
+    {
+      "path": "source/starVLA/model/modules/vlm/QWen3.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/modules/vlm/QWen3.py",
+      "bytes": 7253
+    },
+    {
+      "path": "source/starVLA/model/tools.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/model/tools.py",
+      "bytes": 17659
+    },
+    {
+      "path": "source/starVLA/scoring/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/__init__.py",
+      "bytes": 209
+    },
+    {
+      "path": "source/starVLA/scoring/cheap_proposals.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/cheap_proposals.py",
+      "bytes": 196
+    },
+    {
+      "path": "source/starVLA/scoring/cnf_likelihood.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/cnf_likelihood.py",
+      "bytes": 195
+    },
+    {
+      "path": "source/starVLA/scoring/discrete_flow_likelihood.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/discrete_flow_likelihood.py",
+      "bytes": 205
+    },
+    {
+      "path": "source/starVLA/scoring/flow_residual.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/flow_residual.py",
+      "bytes": 194
+    },
+    {
+      "path": "source/starVLA/scoring/partial_gradient.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/partial_gradient.py",
+      "bytes": 197
+    },
+    {
+      "path": "source/starVLA/scoring/residual_cache.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/residual_cache.py",
+      "bytes": 195
+    },
+    {
+      "path": "source/starVLA/scoring/shuffle.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/shuffle.py",
+      "bytes": 188
+    },
+    {
+      "path": "source/starVLA/scoring/starvla_likelihood.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/starvla_likelihood.py",
+      "bytes": 199
+    },
+    {
+      "path": "source/starVLA/scoring/utility_audit.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/scoring/utility_audit.py",
+      "bytes": 194
+    },
+    {
+      "path": "source/starVLA/training/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/training/__init__.py",
+      "bytes": 0
+    },
+    {
+      "path": "source/starVLA/training/infoflow.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/training/infoflow.py",
+      "bytes": 182
+    },
+    {
+      "path": "source/starVLA/training/train_starvla.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/training/train_starvla.py",
+      "bytes": 99854
+    },
+    {
+      "path": "source/starVLA/training/trainer_utils/__init__.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/training/trainer_utils/__init__.py",
+      "bytes": 44
+    },
+    {
+      "path": "source/starVLA/training/trainer_utils/config_tracker.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/training/trainer_utils/config_tracker.py",
+      "bytes": 21344
+    },
+    {
+      "path": "source/starVLA/training/trainer_utils/monkey_patch.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/training/trainer_utils/monkey_patch.py",
+      "bytes": 19353
+    },
+    {
+      "path": "source/starVLA/training/trainer_utils/overwatch.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/training/trainer_utils/overwatch.py",
+      "bytes": 6848
+    },
+    {
+      "path": "source/starVLA/training/trainer_utils/trainer_tools.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "starVLA/training/trainer_utils/trainer_tools.py",
+      "bytes": 23838
+    },
+    {
+      "path": "source/tests/test_xtrainer2_optr_config.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "tests/test_xtrainer2_optr_config.py",
+      "bytes": 6916
+    },
+    {
+      "path": "source/tests/test_xtrainer2_pack_phone_config.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "tests/test_xtrainer2_pack_phone_config.py",
+      "bytes": 11364
+    },
+    {
+      "path": "source/tests/test_xtrainer2_pick_bottle_config.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "tests/test_xtrainer2_pick_bottle_config.py",
+      "bytes": 4381
+    },
+    {
+      "path": "source/tests/test_xtrainer2_policy_server_contract.py",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "tests/test_xtrainer2_policy_server_contract.py",
+      "bytes": 9812
+    },
+    {
+      "path": "source/third_party/frameskip/LICENSE",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "third_party/frameskip/LICENSE",
+      "bytes": 1387
+    },
+    {
+      "path": "source/UPSTREAM.md",
+      "repository": "miaochengy-pngu/ReTime_source",
+      "ref": "main",
+      "source_path": "UPSTREAM.md",
+      "bytes": 1460
+    }
+  ]
+}

real_robot/source/LICENSE

@@ -0,0 +1,24 @@
+    MIT License
+
+    Copyright (c) StarVLA Team.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    Rebases are allowed for forks and feature branches. When rebasing from upstream StarVLA, use descriptive commit messages, e.g., "chore: clone from StarVLA".
+    Preserve attribution: keep at least the two latest upstream StarVLA commits as separate.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE

real_robot/source/NOTICE

@@ -0,0 +1,40 @@
+ReTime is a research extension of StarVLA, not an official StarVLA release.
+
+StarVLA: https://github.com/starVLA/starVLA
+Base commit: 3422b9f2387b6f682cf02802904a77b23ab13afd
+Copyright (c) StarVLA Team.
+The complete upstream LICENSE is retained, including its additional attribution
+and commit-preservation language. The original base commit and its preceding
+upstream commit remain separate in the supplied Git history.
+
+Modifications in this artifact include temporal sampling/proposals, ordinary
+importance correction, scoring utilities, model/data/trainer integration,
+portable configurations, and the ReTime command-line interface.
+Original ReTime additions in this artifact are distributed under the root
+LICENSE terms. Their authors' identities are withheld during anonymous review.
+Existing upstream author/copyright notices identify the respective third-party
+contributors.
+
+NVIDIA Isaac-GR00T-derived code retains its source notices and Apache-2.0 terms.
+See licenses/Isaac-GR00T-LICENSE. Modified source files carry a modification notice.
+https://github.com/NVIDIA/Isaac-GR00T
+
+OpenVLA/Prismatic-derived logging code retains its attribution.
+See licenses/OpenVLA-LICENSE (MIT).
+https://github.com/openvla/openvla
+
+The Florence wrapper attributes X-VLA-derived code in its source.
+See licenses/X-VLA-LICENSE (Apache-2.0).
+https://github.com/2toinf/X-VLA
+
+The minimal FrameSkip source snapshot retains third_party/frameskip/LICENSE.
+Rotation conversion code in starVLA/dataloader/gr00t_lerobot/transform/state_action.py
+derives from Diffusion Policy. Its MIT license is retained in
+licenses/Diffusion-Policy-LICENSE.
+The adapted msgpack-numpy utility in deployment/model_server/tools/msgpack_numpy.py
+retains the original BSD license in licenses/msgpack-numpy-LICENSE.
+Third-party components remain subject to their respective licenses; the root
+license does not replace these component-specific terms.
+
+No datasets, pretrained model weights, checkpoints, benchmark repositories,
+or robot-client contents are redistributed in the working source tree.

real_robot/source/README.md

@@ -0,0 +1,222 @@
+# ReTime for StarVLA
+
+ReTime is an opt-in temporal-sampling extension for the accompanying StarVLA
+backend. It preserves the dataset/trajectory distribution and changes the
+within-trajectory temporal proposal:
+
+```text
+q(t | e) = (1 - alpha) p(t | e) + alpha r(t | e)
+training loss = mean((p / q) * per_sample_loss)
+```
+
+The weights are ordinary importance weights, not batch-normalized weights.
+`K` controls the frozen-policy scoring grid. `shuffle` permutes the residual
+masses within each trajectory; `uniform` uses the baseline draw stream.
+
+## Install
+
+Use Linux, Python 3.11, a CUDA driver compatible with PyTorch 2.7.1, and the
+dependencies in `requirements.txt`. The core training packages are pinned to
+the versions used in the GPU integration test. From this source tree:
+
+```bash
+python -m pip install -r requirements.txt
+python -m pip install -e .
+retime --help
+```
+
+The supplied Qwen-GR00T configurations additionally use `AdamW8bit` and
+`flash_attention_2`. Install their optional dependencies after the base
+environment and select a Flash Attention build compatible with your CUDA and
+PyTorch installation:
+
+```bash
+python -m pip install -r requirements-qwen.txt
+python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
+```
+
+Flash Attention may require a local CUDA toolkit and build tools. The version
+above was present in the GPU validation environments. Both production Qwen
+continuation configurations use Flash Attention 2. The bounded Qwen Bridge
+integration run used PyTorch SDPA on the same model/data path.
+
+This release supports the supplied Florence/Qwen GR00T configurations and
+LeRobot data layouts. The backend includes the per-example loss and sampler
+integration required by ReTime; it is not a claim of compatibility with every
+unmodified StarVLA release or arbitrary policy architecture.
+
+## Training
+
+The trained common-start checkpoints used in the paper are not redistributed
+with this anonymous artifact. Reviewers should either train their own
+common-start policy using `retime-pretrain` or provide a compatible complete
+checkpoint. The backbone, dataset, and benchmark assets must also be provided
+locally.
+
+Select a configuration matching the policy architecture, action dimensions,
+dataset mixture, and training budget. Matching supplied pairs are:
+
+| Setting | Policy configuration | Start → stop | Global batch | Accelerate configuration |
+| --- | --- | ---: | ---: | --- |
+| Florence × Bridge/RT-1 | `configs/florence_bridge_20k_plus500.yaml` | 20K → 20.5K | 64 | `configs/deepspeed_florence_2gpu.yaml` |
+| Qwen × Bridge/RT-1 | `configs/qwen_bridge_29k_plus500.yaml` | 29K → 29.5K | 48 | `configs/qwen_single_1gpu.yaml` |
+| Florence × RoboTwin | `configs/florence_robotwin_10k_plus500.yaml` | 10K → 10.5K | 64 | `configs/deepspeed_florence_robotwin_2gpu.yaml` |
+| Qwen × RoboTwin | `configs/qwen_robotwin_11k_plus500.yaml` | 11K → 11.5K | 40 | `configs/deepspeed_qwen_robotwin_2gpu.yaml` |
+
+These are the four main simulation continuation contracts. Every arm restores
+the complete shared starting state and performs exactly 500 optimizer updates.
+Bridge runs additionally require `--statistics-path`.
+
+### Train the base policy locally
+
+Reviewers do not need the authors' original common-start checkpoint.
+`retime-pretrain` can train a reviewer-owned common-start policy with the same
+architecture, optimizer, scheduler, batch geometry, and dataset-sampling
+contract as the selected continuation configuration. For Florence Bridge:
+
+```bash
+retime-pretrain \
+  --config configs/florence_bridge_20k_plus500.yaml \
+  --accelerate-config configs/deepspeed_florence_2gpu.yaml \
+  --base-steps 20000 \
+  --vlm-path /path/to/Florence-2-large \
+  --data-root /path/to/OXE_LEROBOT_DATASET \
+  --statistics-path /path/to/dataset_statistics.json \
+  --omit-data-identity-files \
+  --output /path/to/florence-bridge-base
+```
+
+Use `--base-steps 29000`, `10000`, or `11000` with the other three configuration
+pairs. The command builds and saves `manifest.json` before training. Two-process
+configurations produce `checkpoints/steps_<N>_accelerate`; Qwen Bridge produces
+`checkpoints/steps_29000_pytorch_model.pt` with adjacent training-state and
+commit files. Both formats contain the model, optimizer, scheduler, RNG state,
+sampling position, and completed-update counter required by exact continuation.
+
+`--omit-data-identity-files` stores the computed dataset inventory inside the
+manifest. Use the same option for continuation. This avoids requiring separate
+provenance and fingerprint files while still checking the dataset inventory
+against the frozen manifest.
+
+For the Florence Bridge configuration:
+
+```bash
+retime \
+  --config configs/florence_bridge_20k_plus500.yaml \
+  --accelerate-config configs/deepspeed_florence_2gpu.yaml \
+  --base-policy /path/to/steps_20000_accelerate \
+  --start resume \
+  --manifest /path/to/florence-bridge-base/manifest.json \
+  --vlm-path /path/to/Florence-2-large \
+  --data-root /path/to/OXE_LEROBOT_DATASET \
+  --statistics-path /path/to/dataset_statistics.json \
+  --omit-data-identity-files \
+  --alpha 0.2 --K 8 \
+  --mode retime uniform shuffle \
+  --output /path/to/new-run
+```
+
+Choose one mode or several. The command prepares valid-anchor support from the
+dataset, generates the residual proposal once, constructs Shuffle when selected,
+and trains each selected arm sequentially from the same base policy. A
+uniform-only run does not score a proposal. Invalid probabilities,
+incompatible tensor shapes, incomplete
+checkpoint files, and mismatched data support still produce errors.
+
+`--base-policy` denotes the reviewer-provided common-start VLA checkpoint.
+It is not bundled with this anonymous artifact and is not downloaded
+automatically. `--vlm-path` denotes its Florence/Qwen backbone. They are
+distinct inputs. The main configurations use
+`--start resume`, which is required for the reported +500 continuation.
+Florence Bridge and both RoboTwin settings accept a complete Accelerate
+checkpoint directory. Qwen Bridge accepts its committed single-process model
+file together with the adjacent training-state and commit files.
+
+Exact resume restores model, optimizer, scheduler, RNG, sampler position/data
+order, and completed-update counter. Supply the `manifest.json` produced by
+the base-policy run and keep its `config.full.yaml` in the original run layout.
+When separate provenance/fingerprint files are available, omit
+`--omit-data-identity-files` and set the corresponding environment variables in
+both commands. A single-process model file is accepted only when its adjacent
+`*_training_state.pt` and `*_checkpoint.json` files form a complete committed
+checkpoint.
+
+`--start weights` remains available for a deliberate fresh optimizer/scheduler
+run from model weights. It starts at update zero, so it does not reproduce any
+of the four +500 experiments even when a continuation configuration is selected.
+
+The launcher never changes microbatch, world size, accumulation, optimizer,
+schedule, or random seeds to select an arm. The supplied Bridge pairing remains
+`4 x 2 x 8 = 64`. The table above supplies matching Accelerate configurations
+for the other three settings. Select a smaller dataset mixture with `--data-mix`
+when checking a subset; the default RoboTwin configurations cover all 50 tasks.
+
+Add `--dry-run` to validate inputs and print the command plan without writing or
+launching. Successful runs produce `config.yaml`, `plan.json`, a proposal/cache,
+and separate `runs/retime`, `runs/uniform`, and `runs/shuffle` directories. Use a
+fresh output directory for each experiment. Existing proposal caches are bound
+to the checkpoint content and support; incompatible caches cannot be reused.
+
+## Layout
+
+```text
+retime/                  method implementation and command-line entrypoint
+  pretrain.py            reviewer-owned common-start policy training
+  objective.py           ordinary p/q correction
+  sampler.py             deterministic hierarchical draw stream
+  proposal.py            full-support temporal proposal
+  scoring/               frozen-policy scoring and Shuffle control
+  commands/              proposal generation
+starVLA/                 attributed compatible training/model/data backend
+scripts/                 legacy entrypoints and optional analysis utilities
+configs/                 model, data, and training settings
+licenses/                third-party license texts
+```
+
+Legacy `starVLA.*` method imports forward to `retime.*` for compatibility.
+Analysis utilities are optional research tools and are not training prerequisites.
+
+## Compatibility
+
+| Interface | Support |
+| --- | --- |
+| StarVLA training | The original `starVLA/training/train_starvla.py` entrypoint is retained. |
+| Legacy Python imports | Objective, sampler, proposal, and scoring modules forward to ReTime implementations. |
+| Proposal and manifest files | Existing schemas and probability semantics are preserved. |
+| Model initialization | Full model weights initialize a fresh optimizer and scheduler. |
+| Full-state continuation | `--start resume` restores a complete Accelerate directory or committed single-process checkpoint. |
+| Distributed training | Matching one- and two-GPU configurations are supplied for all four settings. |
+
+Florence-GR00T and Qwen-GR00T were exercised on GPUs with both Bridge/RT-1 and
+RoboTwin data. Other policy architectures require a compatible per-sample loss
+and dataset adapter.
+Historical `OPTR_*` environment variables and `InfoFlow` module names in
+example configurations are compatibility identifiers for existing datasets and
+checkpoints. The supported command-line entrypoint for this artifact is `retime`.
+
+## Tests and scope
+
+```bash
+python -m pip install pytest
+python -m pytest tests/test_retime_plugin.py -q
+```
+
+These CPU tests check proposal generation, objective correction, deterministic
+sampling, Shuffle invariants, legacy imports, CLI dispatch, and error propagation.
+A separate GPU integration test completed proposal generation, two optimizer
+updates, checkpoint saving, and strict model reload for each of the three modes
+in all four settings. See `VALIDATION.md` for details.
+Datasets, trained weights, checkpoints, and benchmark repositories are not
+redistributed in the working source tree.
+
+## Attribution and anonymous review
+
+Built on [StarVLA](https://github.com/starVLA/starVLA), pinned to
+`3422b9f2387b6f682cf02802904a77b23ab13afd`. See `UPSTREAM.md`, `LICENSE`, and
+`NOTICE`. Retain the complete license, source copyright notices, and the
+separate original upstream commits in the accompanying Git history.
+
+Third-party attribution is intentionally visible. The anonymous additions have
+no author-account remote or personal development history. Publish through an
+anonymous review endpoint, not an author's personal repository URL. Avoid
+uploading local training outputs: they can contain machine paths and metadata.

real_robot/source/UPSTREAM.md

@@ -0,0 +1,29 @@
+# Upstream provenance
+
+- Repository: https://github.com/starVLA/starVLA
+- Original branch: `starVLA`
+- Base: `3422b9f2387b6f682cf02802904a77b23ab13afd`
+- Preceding upstream commit: `61ea14279a22739316f9c1ca74c18d535247fe05`
+- Base LICENSE SHA256: `6ca01970c4e66d2d5bebd0a00b55de5b4081017e0f0c97ddb415cda28e3e8ffa`
+
+The base revision was recovered from the original development snapshot's
+provenance record and checked against the public upstream Git object and license.
+The two original upstream commits are retained unchanged. A separate anonymous
+commit contains the review artifact and modifications. Personal development
+commits and remotes are not part of this distribution.
+
+The source tree is a selected, modified research backend, not an unmodified
+checkout of that revision. Per-file notices and NOTICE describe the derivative
+work. Do not replace LICENSE with a generic MIT template: this upstream text
+contains additional attribution/commit-preservation language.
+
+The distribution ZIP includes `review-history.bundle`, containing the public
+upstream ancestry and the anonymous review commit. Restore a Git repository with:
+
+```bash
+git clone --branch anonymous-review review-history.bundle retime-review
+```
+
+This preserves the original upstream commit objects and creates no relationship
+to the authors' personal development repository. Use the restored history when
+publishing to an anonymous repository; do not squash away the upstream commits.

real_robot/source/configs/deepspeed_florence_8gpu.yaml

@@ -0,0 +1,23 @@
+compute_environment: LOCAL_MACHINE
+debug: false
+deepspeed_config:
+  gradient_accumulation_steps: 4
+  offload_optimizer_device: none
+  offload_param_device: none
+  zero3_init_flag: true
+  # Required: _save_accelerate_checkpoint() calls accelerator.get_state_dict(),
+  # which raises under ZeRO-3 unless the 16-bit weights are gathered on save.
+  # Without this the run trains fine and then dies at the first save_steps entry.
+  zero3_save_16bit_model: true
+  zero_stage: 3
+distributed_type: DEEPSPEED
+downcast_bf16: "no"
+enable_cpu_affinity: false
+machine_rank: 0
+main_training_function: main
+mixed_precision: bf16
+num_machines: 1
+num_processes: 8
+rdzv_backend: static
+same_network: true
+use_cpu: false

real_robot/source/configs/florence_xtrainer2_pack_phone_residual_20ep_to_60ep.yaml

@@ -0,0 +1,101 @@
+run_id: florence-xtrainer2-pack-phone-residual-20ep-to-60ep
+run_root_dir: ${oc.env:OPTR_RUN_ROOT}
+seed: 117465
+wandb_entity: null
+wandb_project: OPTR-Florence-Xtrainer2
+wandb_group: pack-phone-optr
+wandb_job_type: train
+wandb_tags: [florence, xtrainer2, pack-phone, optr, residual, 20ep-common-start]
+is_debug: false
+version_id: "0.21"
+
+framework:
+  name: QwenGR00T
+  qwenvl:
+    base_vlm: ${oc.env:FLORENCE_MODEL_PATH}
+    attn_implementation: eager
+    vl_hidden_dim: 1024
+    selected_view_indices: [0, 1, 2]
+    compile_vision_tower: false
+  action_model:
+    action_model_type: DiT-B
+    hidden_size: 1024
+    add_pos_embed: true
+    max_seq_len: 1024
+    action_dim: 20
+    state_dim: 20
+    action_horizon: 16
+    future_action_window_size: 15
+    noise_beta_alpha: 1.5
+    noise_beta_beta: 1.0
+    noise_s: 0.999
+    num_timestep_buckets: 1000
+    num_inference_timesteps: 4
+    num_target_vision_tokens: 32
+    diffusion_model_cfg:
+      cross_attention_dim: 1024
+      dropout: 0.2
+      final_dropout: true
+      interleave_self_attention: true
+      norm_type: ada_norm
+      num_layers: 16
+      output_dim: 1024
+      positional_embeddings: null
+
+datasets:
+  vla_data:
+    dataset_py: lerobot_datasets
+    data_root_dir: ${oc.env:XTRAINER2_DATA_ROOT}
+    data_mix: xtrainer2_pack_phone
+    action_type: abs_eef
+    action_mode: abs
+    include_state: true
+    per_device_batch_size: 4
+    image_size: [224, 224]
+    video_backend: pyav
+    num_workers: 8
+    expected_world_size: 8
+    traceable_sampling: true
+    dataset_provenance_path: ${oc.env:XTRAINER2_PACK_PHONE_PROVENANCE_PATH}
+    dataset_content_fingerprint_path: ${oc.env:XTRAINER2_PACK_PHONE_FINGERPRINT_PATH}
+    valid_anchor_manifest_path: ${oc.env:XTRAINER2_PACK_PHONE_MANIFEST_PATH}
+    temporal_proposal_path: ${oc.env:OPTR_TEMPORAL_PROPOSAL_PATH}
+    transform_rng_seed: 298226
+
+trainer:
+  use_deepspeed: true
+  max_train_steps: 52380
+  num_warmup_steps: 5000
+  save_interval: null
+  save_steps: [26190, 34920, 43650, 52380]
+  eval_interval: null
+  repeated_diffusion_steps: 4
+  pretrained_checkpoint: null
+  is_resume: true
+  allow_model_only_resume: false
+  resume_accelerate_checkpoint: ${oc.env:XTRAINER2_PACK_PHONE_RESUME_DIR}
+  branch_parent_seed: 42
+  branch_rng_seed: 117465
+  model_rng_seed: 588643
+  stop_after_steps: 52380
+  preflight_validate_first_batch: true
+  learning_rate:
+    base: 1.0e-05
+    qwen_vl_interface: 1.0e-05
+    action_model: 1.0e-04
+  lr_scheduler_type: cosine_with_min_lr
+  scheduler_specific_kwargs:
+    min_lr: 5.0e-07
+  freeze_modules: ""
+  trainable_modules: ""
+  frozen_modules_eval: false
+  logging_frequency: 50
+  gradient_clipping: 1.0
+  gradient_accumulation_steps: 4
+  save_format: pt
+  save_final_checkpoint: false
+  optimizer:
+    name: AdamW
+    betas: [0.9, 0.95]
+    eps: 1.0e-08
+    weight_decay: 1.0e-08

real_robot/source/configs/florence_xtrainer2_pack_phone_residual_resume_smoke.yaml

@@ -0,0 +1,104 @@
+# External exact-resume smoke. It uses the production batch geometry and
+# common-start, but writes to an independent run and performs at most two
+# optimizer updates.
+run_id: florence-xtrainer2-pack-phone-residual-resume-smoke
+run_root_dir: ${oc.env:OPTR_RUN_ROOT}
+seed: 117465
+wandb_entity: null
+wandb_project: OPTR-Florence-Xtrainer2
+wandb_group: pack-phone-optr
+wandb_job_type: smoke
+wandb_tags: [florence, xtrainer2, pack-phone, residual, resume-smoke]
+is_debug: false
+version_id: "0.21"
+
+framework:
+  name: QwenGR00T
+  qwenvl:
+    base_vlm: ${oc.env:FLORENCE_MODEL_PATH}
+    attn_implementation: eager
+    vl_hidden_dim: 1024
+    selected_view_indices: [0, 1, 2]
+    compile_vision_tower: false
+  action_model:
+    action_model_type: DiT-B
+    hidden_size: 1024
+    add_pos_embed: true
+    max_seq_len: 1024
+    action_dim: 20
+    state_dim: 20
+    action_horizon: 16
+    future_action_window_size: 15
+    noise_beta_alpha: 1.5
+    noise_beta_beta: 1.0
+    noise_s: 0.999
+    num_timestep_buckets: 1000
+    num_inference_timesteps: 4
+    num_target_vision_tokens: 32
+    diffusion_model_cfg:
+      cross_attention_dim: 1024
+      dropout: 0.2
+      final_dropout: true
+      interleave_self_attention: true
+      norm_type: ada_norm
+      num_layers: 16
+      output_dim: 1024
+      positional_embeddings: null
+
+datasets:
+  vla_data:
+    dataset_py: lerobot_datasets
+    data_root_dir: ${oc.env:XTRAINER2_DATA_ROOT}
+    data_mix: xtrainer2_pack_phone
+    action_type: abs_eef
+    action_mode: abs
+    include_state: true
+    per_device_batch_size: 4
+    image_size: [224, 224]
+    video_backend: pyav
+    num_workers: 8
+    expected_world_size: 8
+    traceable_sampling: true
+    dataset_provenance_path: ${oc.env:XTRAINER2_PACK_PHONE_PROVENANCE_PATH}
+    dataset_content_fingerprint_path: ${oc.env:XTRAINER2_PACK_PHONE_FINGERPRINT_PATH}
+    valid_anchor_manifest_path: ${oc.env:XTRAINER2_PACK_PHONE_MANIFEST_PATH}
+    temporal_proposal_path: ${oc.env:OPTR_TEMPORAL_PROPOSAL_PATH}
+    transform_rng_seed: 298226
+
+trainer:
+  use_deepspeed: true
+  max_train_steps: 52380
+  num_warmup_steps: 5000
+  save_interval: null
+  save_steps: []
+  eval_interval: null
+  repeated_diffusion_steps: 4
+  pretrained_checkpoint: null
+  is_resume: true
+  allow_model_only_resume: false
+  resume_accelerate_checkpoint: ${oc.env:XTRAINER2_PACK_PHONE_RESUME_DIR}
+  branch_parent_seed: 42
+  branch_rng_seed: 117465
+  model_rng_seed: 588643
+  stop_after_steps: 17462
+  preflight_validate_first_batch: true
+  learning_rate:
+    base: 1.0e-05
+    qwen_vl_interface: 1.0e-05
+    action_model: 1.0e-04
+  lr_scheduler_type: cosine_with_min_lr
+  scheduler_specific_kwargs:
+    min_lr: 5.0e-07
+  freeze_modules: ""
+  trainable_modules: ""
+  frozen_modules_eval: false
+  logging_frequency: 1
+  gradient_clipping: 1.0
+  gradient_accumulation_steps: 4
+  save_format: pt
+  save_final_checkpoint: false
+  optimizer:
+    name: AdamW
+    betas: [0.9, 0.95]
+    eps: 1.0e-08
+    weight_decay: 1.0e-08

real_robot/source/configs/florence_xtrainer2_pack_phone_smoke.yaml

@@ -0,0 +1,117 @@
+# Smoke test for the pack_phone uniform run. Six optimizer steps with a
+# checkpoint at step 4, which exercises the ZeRO-3 gather-and-consolidate save
+# path that would otherwise only be reached after ~10 epochs.
+# Keeps per_device_batch_size, accumulation, and world size identical to the
+# real config so memory and the global batch of 128 are representative.
+# vision-tower compile is off here to keep startup fast; the real run enables it.
+run_id: florence-xtrainer2-pack-phone-smoke
+run_root_dir: ${oc.env:OPTR_RUN_ROOT}
+seed: 42
+wandb_entity: null
+wandb_project: OPTR-Florence-Xtrainer2
+wandb_group: pack-phone
+wandb_job_type: smoke
+wandb_tags:
+  - florence
+  - xtrainer2
+  - pack-phone
+  - smoke
+is_debug: false
+version_id: "0.21"
+
+framework:
+  name: QwenGR00T
+  qwenvl:
+    # NOTE: the backbone is selected by substring match on this path
+    # (starVLA/model/modules/vlm/__init__.py). The resolved value MUST contain
+    # "florence" or startup fails with NotImplementedError.
+    base_vlm: ${oc.env:FLORENCE_MODEL_PATH}
+    attn_implementation: eager
+    vl_hidden_dim: 1024
+    selected_view_indices: [0, 1, 2]
+    compile_vision_tower: false
+  action_model:
+    action_model_type: DiT-B
+    hidden_size: 1024
+    add_pos_embed: true
+    max_seq_len: 1024
+    action_dim: 20
+    state_dim: 20
+    action_horizon: 16
+    future_action_window_size: 15
+    # NOTE: the effective flow-matching repeat factor is trainer.repeated_diffusion_steps.
+    # A framework-level key here would be silently ignored; do not re-add it.
+    noise_beta_alpha: 1.5
+    noise_beta_beta: 1.0
+    noise_s: 0.999
+    num_timestep_buckets: 1000
+    num_inference_timesteps: 4
+    num_target_vision_tokens: 32
+    diffusion_model_cfg:
+      cross_attention_dim: 1024
+      dropout: 0.2
+      final_dropout: true
+      interleave_self_attention: true
+      norm_type: ada_norm
+      num_layers: 16
+      output_dim: 1024
+      positional_embeddings: null
+
+datasets:
+  vla_data:
+    dataset_py: lerobot_datasets
+    data_root_dir: ${oc.env:XTRAINER2_DATA_ROOT}
+    data_mix: xtrainer2_pack_phone
+    action_type: abs_eef
+    action_mode: abs
+    include_state: true
+    # 4 x 8 GPUs x accum 4 = global batch 128. Measured peaks on idle 80 GB
+    # cards: batch 4 = 60.3 GB, batch 5 = 73.4 GB, batch 6 OOMs. Holding
+    # global batch at 128 also requires per_device * accum == 16, so 4/4 is
+    # the largest usable microbatch; 8/2 does not fit in memory.
+    # See scripts/probe_batch_size.sh.
+    per_device_batch_size: 4
+    image_size: [224, 224]
+    video_backend: pyav
+    num_workers: 2
+    expected_world_size: 8
+    traceable_sampling: true
+    dataset_provenance_path: ${oc.env:XTRAINER2_PACK_PHONE_PROVENANCE_PATH}
+    dataset_content_fingerprint_path: ${oc.env:XTRAINER2_PACK_PHONE_FINGERPRINT_PATH}
+    valid_anchor_manifest_path: null
+    temporal_proposal_path: null
+
+trainer:
+  use_deepspeed: true
+  max_train_steps: 6
+  num_warmup_steps: 2
+  save_interval: null
+  save_steps: [4]
+  eval_interval: null
+  # Effective flow-matching repeat factor (framework.action_model has no say).
+  # Per-DiT micro-batch is per_device_batch_size * this = 4 * 4 = 16.
+  repeated_diffusion_steps: 4
+  learning_rate:
+    base: 1.0e-05
+    qwen_vl_interface: 1.0e-05
+    action_model: 1.0e-04
+  lr_scheduler_type: cosine_with_min_lr
+  scheduler_specific_kwargs:
+    min_lr: 5.0e-07
+  freeze_modules: ""
+  trainable_modules: ""
+  frozen_modules_eval: false
+  logging_frequency: 1
+  gradient_clipping: 1.0
+  # MUST equal deepspeed_config.gradient_accumulation_steps in the accelerate
+  # yaml. DeepSpeed's value governs the optimizer; this one sizes the dataloader
+  # truncation. A mismatch is only a warning, so keep them in sync by hand.
+  # Global batch = per_device 4 * 8 GPUs * accum 4 = 128.
+  gradient_accumulation_steps: 4
+  save_format: pt
+  save_final_checkpoint: false
+  optimizer:
+    name: AdamW
+    betas: [0.9, 0.95]
+    eps: 1.0e-08
+    weight_decay: 1.0e-08

real_robot/source/configs/florence_xtrainer2_pack_phone_uniform.yaml

@@ -0,0 +1,116 @@
+run_id: florence-xtrainer2-pack-phone-uniform
+run_root_dir: ${oc.env:OPTR_RUN_ROOT}
+seed: 42
+wandb_entity: null
+wandb_project: OPTR-Florence-Xtrainer2
+wandb_group: pack-phone
+wandb_job_type: train
+wandb_tags:
+  - florence
+  - xtrainer2
+  - pack-phone
+  - train
+is_debug: false
+version_id: "0.21"
+
+framework:
+  name: QwenGR00T
+  qwenvl:
+    # NOTE: the backbone is selected by substring match on this path
+    # (starVLA/model/modules/vlm/__init__.py). The resolved value MUST contain
+    # "florence" or startup fails with NotImplementedError.
+    base_vlm: ${oc.env:FLORENCE_MODEL_PATH}
+    attn_implementation: eager
+    vl_hidden_dim: 1024
+    selected_view_indices: [0, 1, 2]
+    # Must stay false: torch.compile with mode="reduce-overhead" (CUDA graphs,
+    # Florence2.py:150-156) fails under ZeRO-3 with
+    # "AssertionError: wrong number of dimensions" from cudagraph_trees'
+    # assert_size_stride, because gathered parameter shapes differ between the
+    # trace and the replay. Verified failing on 8 ranks.
+    compile_vision_tower: false
+  action_model:
+    action_model_type: DiT-B
+    hidden_size: 1024
+    add_pos_embed: true
+    max_seq_len: 1024
+    action_dim: 20
+    state_dim: 20
+    action_horizon: 16
+    future_action_window_size: 15
+    # NOTE: the effective flow-matching repeat factor is trainer.repeated_diffusion_steps.
+    # A framework-level key here would be silently ignored; do not re-add it.
+    noise_beta_alpha: 1.5
+    noise_beta_beta: 1.0
+    noise_s: 0.999
+    num_timestep_buckets: 1000
+    num_inference_timesteps: 4
+    num_target_vision_tokens: 32
+    diffusion_model_cfg:
+      cross_attention_dim: 1024
+      dropout: 0.2
+      final_dropout: true
+      interleave_self_attention: true
+      norm_type: ada_norm
+      num_layers: 16
+      output_dim: 1024
+      positional_embeddings: null
+
+datasets:
+  vla_data:
+    dataset_py: lerobot_datasets
+    data_root_dir: ${oc.env:XTRAINER2_DATA_ROOT}
+    data_mix: xtrainer2_pack_phone
+    action_type: abs_eef
+    action_mode: abs
+    include_state: true
+    # 4 x 8 GPUs x accum 4 = global batch 128. Measured peaks on idle 80 GB
+    # cards: batch 4 = 60.3 GB, batch 5 = 73.4 GB, batch 6 OOMs. Holding
+    # global batch at 128 also requires per_device * accum == 16, so 4/4 is
+    # the largest usable microbatch; 8/2 does not fit in memory.
+    # See scripts/probe_batch_size.sh.
+    per_device_batch_size: 4
+    image_size: [224, 224]
+    video_backend: pyav
+    num_workers: 8
+    expected_world_size: 8
+    traceable_sampling: true
+    dataset_provenance_path: ${oc.env:XTRAINER2_PACK_PHONE_PROVENANCE_PATH}
+    dataset_content_fingerprint_path: ${oc.env:XTRAINER2_PACK_PHONE_FINGERPRINT_PATH}
+    valid_anchor_manifest_path: null
+    temporal_proposal_path: null
+
+trainer:
+  use_deepspeed: true
+  max_train_steps: 52380
+  num_warmup_steps: 5000
+  save_interval: null
+  save_steps: [8730, 17460, 26190, 34920, 43650, 52380]
+  eval_interval: null
+  # Effective flow-matching repeat factor (framework.action_model has no say).
+  # Per-DiT micro-batch is per_device_batch_size * this = 4 * 4 = 16.
+  repeated_diffusion_steps: 4
+  learning_rate:
+    base: 1.0e-05
+    qwen_vl_interface: 1.0e-05
+    action_model: 1.0e-04
+  lr_scheduler_type: cosine_with_min_lr
+  scheduler_specific_kwargs:
+    min_lr: 5.0e-07
+  freeze_modules: ""
+  trainable_modules: ""
+  frozen_modules_eval: false
+  logging_frequency: 50
+  gradient_clipping: 1.0
+  # MUST equal deepspeed_config.gradient_accumulation_steps in the accelerate
+  # yaml. DeepSpeed's value governs the optimizer; this one sizes the dataloader
+  # truncation. A mismatch is only a warning, so keep them in sync by hand.
+  # Global batch = per_device 4 * 8 GPUs * accum 4 = 128.
+  gradient_accumulation_steps: 4
+  save_format: pt
+  save_final_checkpoint: false
+  optimizer:
+    name: AdamW
+    betas: [0.9, 0.95]
+    eps: 1.0e-08
+    weight_decay: 1.0e-08

real_robot/source/configs/florence_xtrainer2_pick_bottle_rel_aug_smoke.yaml

@@ -0,0 +1,119 @@
+# Smoke test for pick_bottle rel-EE + image-aug. Six optimizer steps with a
+# checkpoint at step 4, which exercises relative-action statistics, random
+# crop, and the ZeRO-3 gather-and-consolidate save path.
+run_id: florence-xtrainer2-pick-bottle-rel-aug-smoke
+run_root_dir: ${oc.env:OPTR_RUN_ROOT}
+seed: 42
+wandb_entity: null
+wandb_project: OPTR-Florence-Xtrainer2
+wandb_group: pick-bottle
+wandb_job_type: smoke
+wandb_tags:
+  - florence
+  - xtrainer2
+  - pick-bottle
+  - rel-eef
+  - image-aug
+  - smoke
+is_debug: false
+version_id: "0.21"
+
+framework:
+  name: QwenGR00T
+  qwenvl:
+    # NOTE: the backbone is selected by substring match on this path
+    # (starVLA/model/modules/vlm/__init__.py). The resolved value MUST contain
+    # "florence" or startup fails with NotImplementedError.
+    base_vlm: ${oc.env:FLORENCE_MODEL_PATH}
+    attn_implementation: eager
+    vl_hidden_dim: 1024
+    selected_view_indices: [0, 1, 2]
+    compile_vision_tower: false
+  action_model:
+    action_model_type: DiT-B
+    hidden_size: 1024
+    add_pos_embed: true
+    max_seq_len: 1024
+    action_dim: 20
+    state_dim: 20
+    action_horizon: 16
+    future_action_window_size: 15
+    # NOTE: the effective flow-matching repeat factor is trainer.repeated_diffusion_steps.
+    # A framework-level key here would be silently ignored; do not re-add it.
+    noise_beta_alpha: 1.5
+    noise_beta_beta: 1.0
+    noise_s: 0.999
+    num_timestep_buckets: 1000
+    num_inference_timesteps: 4
+    num_target_vision_tokens: 32
+    diffusion_model_cfg:
+      cross_attention_dim: 1024
+      dropout: 0.2
+      final_dropout: true
+      interleave_self_attention: true
+      norm_type: ada_norm
+      num_layers: 16
+      output_dim: 1024
+      positional_embeddings: null
+
+datasets:
+  vla_data:
+    dataset_py: lerobot_datasets
+    data_root_dir: ${oc.env:XTRAINER2_DATA_ROOT}
+    data_mix: xtrainer2_pick_bottle
+    action_type: rel_eef
+    action_mode: rel
+    action_mode_apply_keys:
+      - action.left_position
+      - action.left_rotation
+      - action.right_position
+      - action.right_rotation
+    include_state: true
+    # 4 x 8 GPUs x accum 4 = global batch 128. Same memory envelope as the
+    # production run.
+    per_device_batch_size: 4
+    image_size: [224, 224]
+    video_backend: pyav
+    num_workers: 2
+    expected_world_size: 8
+    traceable_sampling: true
+    transform_rng_seed: 298226
+    dataset_provenance_path: ${oc.env:XTRAINER2_PICK_BOTTLE_PROVENANCE_PATH}
+    dataset_content_fingerprint_path: ${oc.env:XTRAINER2_PICK_BOTTLE_FINGERPRINT_PATH}
+    valid_anchor_manifest_path: null
+    temporal_proposal_path: null
+
+trainer:
+  use_deepspeed: true
+  max_train_steps: 6
+  num_warmup_steps: 2
+  save_interval: null
+  save_steps: [4]
+  eval_interval: null
+  # Effective flow-matching repeat factor (framework.action_model has no say).
+  # Per-DiT micro-batch is per_device_batch_size * this = 4 * 4 = 16.
+  repeated_diffusion_steps: 4
+  learning_rate:
+    base: 1.0e-05
+    qwen_vl_interface: 1.0e-05
+    action_model: 1.0e-04
+  lr_scheduler_type: cosine_with_min_lr
+  scheduler_specific_kwargs:
+    min_lr: 5.0e-07
+  freeze_modules: ""
+  trainable_modules: ""
+  frozen_modules_eval: false
+  logging_frequency: 1
+  gradient_clipping: 1.0
+  # MUST equal deepspeed_config.gradient_accumulation_steps in the accelerate
+  # yaml. DeepSpeed's value governs the optimizer; this one sizes the dataloader
+  # truncation. A mismatch is only a warning, so keep them in sync by hand.
+  # Global batch = per_device 4 * 8 GPUs * accum 4 = 128.
+  gradient_accumulation_steps: 4
+  save_format: pt
+  save_final_checkpoint: false
+  optimizer:
+    name: AdamW
+    betas: [0.9, 0.95]
+    eps: 1.0e-08
+    weight_decay: 1.0e-08

real_robot/source/configs/florence_xtrainer2_pick_bottle_rel_aug_uniform.yaml

@@ -0,0 +1,123 @@
+run_id: florence-xtrainer2-pick-bottle-rel-aug-uniform
+run_root_dir: ${oc.env:OPTR_RUN_ROOT}
+seed: 42
+wandb_entity: null
+wandb_project: OPTR-Florence-Xtrainer2
+wandb_group: pick-bottle
+wandb_job_type: train
+wandb_tags:
+  - florence
+  - xtrainer2
+  - pick-bottle
+  - rel-eef
+  - image-aug
+  - train
+is_debug: false
+version_id: "0.21"
+
+framework:
+  name: QwenGR00T
+  qwenvl:
+    # NOTE: the backbone is selected by substring match on this path
+    # (starVLA/model/modules/vlm/__init__.py). The resolved value MUST contain
+    # "florence" or startup fails with NotImplementedError.
+    base_vlm: ${oc.env:FLORENCE_MODEL_PATH}
+    attn_implementation: eager
+    vl_hidden_dim: 1024
+    selected_view_indices: [0, 1, 2]
+    # Must stay false: torch.compile with mode="reduce-overhead" (CUDA graphs,
+    # Florence2.py:150-156) fails under ZeRO-3 with
+    # "AssertionError: wrong number of dimensions" from cudagraph_trees'
+    # assert_size_stride, because gathered parameter shapes differ between the
+    # trace and the replay. Verified failing on 8 ranks.
+    compile_vision_tower: false
+  action_model:
+    action_model_type: DiT-B
+    hidden_size: 1024
+    add_pos_embed: true
+    max_seq_len: 1024
+    action_dim: 20
+    state_dim: 20
+    action_horizon: 16
+    future_action_window_size: 15
+    # NOTE: the effective flow-matching repeat factor is trainer.repeated_diffusion_steps.
+    # A framework-level key here would be silently ignored; do not re-add it.
+    noise_beta_alpha: 1.5
+    noise_beta_beta: 1.0
+    noise_s: 0.999
+    num_timestep_buckets: 1000
+    num_inference_timesteps: 4
+    num_target_vision_tokens: 32
+    diffusion_model_cfg:
+      cross_attention_dim: 1024
+      dropout: 0.2
+      final_dropout: true
+      interleave_self_attention: true
+      norm_type: ada_norm
+      num_layers: 16
+      output_dim: 1024
+      positional_embeddings: null
+
+datasets:
+  vla_data:
+    dataset_py: lerobot_datasets
+    data_root_dir: ${oc.env:XTRAINER2_DATA_ROOT}
+    data_mix: xtrainer2_pick_bottle
+    action_type: rel_eef
+    action_mode: rel
+    # Gripper stays absolute; xyz subtracts the current EE; quaternions use
+    # q_rel = inv(q_s0) ⊗ q_a rather than component-wise subtraction.
+    action_mode_apply_keys:
+      - action.left_position
+      - action.left_rotation
+      - action.right_position
+      - action.right_rotation
+    include_state: true
+    # 4 x 8 GPUs x accum 4 = global batch 128. Same memory envelope as pack_phone.
+    per_device_batch_size: 4
+    image_size: [224, 224]
+    video_backend: pyav
+    num_workers: 8
+    expected_world_size: 8
+    traceable_sampling: true
+    transform_rng_seed: 298226
+    dataset_provenance_path: ${oc.env:XTRAINER2_PICK_BOTTLE_PROVENANCE_PATH}
+    dataset_content_fingerprint_path: ${oc.env:XTRAINER2_PICK_BOTTLE_FINGERPRINT_PATH}
+    valid_anchor_manifest_path: null
+    temporal_proposal_path: null
+
+trainer:
+  use_deepspeed: true
+  # 75758 frames / global batch 128 ≈ 591.86 → 592 optimizer steps per data epoch.
+  max_train_steps: 35520
+  num_warmup_steps: 3500
+  save_interval: null
+  save_steps: [5920, 11840, 17760, 23680, 29600, 35520]
+  eval_interval: null
+  # Effective flow-matching repeat factor (framework.action_model has no say).
+  # Per-DiT micro-batch is per_device_batch_size * this = 4 * 4 = 16.
+  repeated_diffusion_steps: 4
+  learning_rate:
+    base: 1.0e-05
+    qwen_vl_interface: 1.0e-05
+    action_model: 1.0e-04
+  lr_scheduler_type: cosine_with_min_lr
+  scheduler_specific_kwargs:
+    min_lr: 5.0e-07
+  freeze_modules: ""
+  trainable_modules: ""
+  frozen_modules_eval: false
+  logging_frequency: 50
+  gradient_clipping: 1.0
+  # MUST equal deepspeed_config.gradient_accumulation_steps in the accelerate
+  # yaml. DeepSpeed's value governs the optimizer; this one sizes the dataloader
+  # truncation. A mismatch is only a warning, so keep them in sync by hand.
+  # Global batch = per_device 4 * 8 GPUs * accum 4 = 128.
+  gradient_accumulation_steps: 4
+  save_format: pt
+  save_final_checkpoint: false
+  optimizer:
+    name: AdamW
+    betas: [0.9, 0.95]
+    eps: 1.0e-08
+    weight_decay: 1.0e-08

real_robot/source/deployment/model_server/policy_norm_processor.py

@@ -0,0 +1,538 @@
+# Copyright 2025 starVLA community. All rights reserved.
+# Licensed under the MIT License.
+"""PolicyNormProcessor — reuse the training-time ComposedModalityTransform.
+
+This class replaces the hand-rolled un-normalization math that previously
+lived in every eval client. It rebuilds the *exact* ``ComposedModalityTransform``
+used at training time from a checkpoint:
+
+  1. Read ``config.yaml`` next to the ckpt → resolve ``data_mix`` →
+     look up ``robot_type`` from ``DATASET_NAMED_MIXTURES`` →
+     fetch the ``DataConfig`` from ``ROBOT_TYPE_CONFIG_MAP``.
+  2. Build the transform pipeline via ``data_config.transform()``.
+  3. Reconstruct a ``DatasetMetadata`` from the saved
+     ``dataset_statistics.json`` (which stores **combined** per-modality
+     arrays of length ``D``) by splitting it into per-key entries that match
+     ``data_config.action_keys`` / ``state_keys``.
+  4. ``set_metadata(...)`` binds the metadata into every transform.
+
+After construction the caller simply invokes :meth:`unapply_actions` /
+:meth:`apply_state` to get/normalize tensors using the same code path as
+training — there is no second source of truth for normalization math.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any, Dict, List, Optional, Sequence
+
+import numpy as np
+import torch
+
+from starVLA.dataloader.gr00t_lerobot.registry import (
+    DATASET_NAMED_MIXTURES,
+    ROBOT_TYPE_CONFIG_MAP,
+)
+from starVLA.dataloader.gr00t_lerobot.schema import (
+    DatasetMetadata,
+    StateActionMetadata,
+)
+from starVLA.dataloader.gr00t_lerobot.transform.base import ComposedModalityTransform
+from starVLA.model.framework.share_tools import read_mode_config
+
+logger = logging.getLogger(__name__)
+
+
+def _resolve_robot_type(
+    model_cfg: dict,
+    unnorm_key: Optional[str] = None,
+) -> str:
+    """Look up the training robot_type from the saved cfg.
+
+    Convention used by starVLA train scripts:
+      ``cfg.datasets.vla_data.data_mix`` is a key in
+      ``DATASET_NAMED_MIXTURES`` whose value is a list of
+      ``(dataset_name, weight, robot_type)`` tuples.
+
+    When a data_mix contains entries from multiple robot types (e.g.
+    ``bridge_rt_1`` covers ``oxe_bridge`` + ``oxe_rt1``), ``unnorm_key``
+    is used to identify which embodiment is requested.  In those mixtures
+    the ``robot_type`` field of each entry **matches** the top-level key in
+    ``dataset_statistics.json``, so ``unnorm_key`` serves as the selector.
+    """
+    try:
+        data_mix = model_cfg["datasets"]["vla_data"]["data_mix"]
+    except (KeyError, TypeError) as e:
+        raise KeyError(
+            "ckpt config.yaml is missing `datasets.vla_data.data_mix`; "
+            "cannot resolve training-time robot_type."
+        ) from e
+
+    if data_mix not in DATASET_NAMED_MIXTURES:
+        raise KeyError(
+            f"data_mix={data_mix!r} not in DATASET_NAMED_MIXTURES "
+            f"(available: {sorted(DATASET_NAMED_MIXTURES.keys())[:20]} ...). "
+            "Did you forget to register the example under examples/<bench>/train_files/data_registry/?"
+        )
+
+    mixture = DATASET_NAMED_MIXTURES[data_mix]
+    robot_types = sorted({entry[2] for entry in mixture})
+
+    if len(robot_types) == 1:
+        return robot_types[0]
+
+    # Multiple robot types in the mixture.
+    # Use unnorm_key as a direct selector: for multi-robot mixtures the
+    # dataset_statistics.json top-level keys equal the robot_type values.
+    if unnorm_key is not None and unnorm_key in robot_types:
+        return unnorm_key
+
+    raise ValueError(
+        f"data_mix={data_mix!r} contains multiple robot_types {robot_types}. "
+        "Pass `unnorm_key` matching one of them to disambiguate "
+        f"(e.g. unnorm_key={robot_types[0]!r})."
+    )
+
+
+def _infer_key_dims(
+    data_config: Any,
+    combined_stats: Dict[str, Any],
+    modality_keys: Sequence[str],
+    modality: str,
+) -> Dict[str, int]:
+    """Compute per-key dimensions for splitting combined stats arrays.
+
+    Lookup priority:
+      1. ``data_config.<modality>_key_dims`` — explicit dict classvar on the DataConfig
+         (required for DataConfigs with non-uniform or multi-dim keys).
+      2. Infer uniformly from stats array length: if ``D_total / n_keys`` is an
+         integer, use that as the uniform per-key dim.
+      3. Fall back to dim=1 when no stats are available (empty combined dict).
+
+    Args:
+        data_config: The DataConfig instance for the current robot type.
+        combined_stats: The raw stats dict for the chosen unnorm_key (the value
+            of ``norm_stats[unnorm_key]`` from ``dataset_statistics.json``).
+        modality_keys: Ordered list of full keys for this modality
+            (e.g. ``["action.left_joints", ...]``).
+        modality: ``"action"`` or ``"state"``.
+
+    Returns:
+        Dict mapping each full key to its integer dimension.
+    """
+    attr = f"{modality}_key_dims"
+    if hasattr(data_config, attr):
+        return dict(getattr(data_config, attr))
+
+    combined = combined_stats.get(modality, {})
+    stat_arr = next((v for k, v in combined.items() if k != "mask"), None)
+    n_keys = len(modality_keys)
+    if stat_arr is not None and n_keys > 0:
+        D_total = len(stat_arr)
+        if D_total == n_keys:
+            return {k: 1 for k in modality_keys}
+        elif n_keys > 0 and D_total % n_keys == 0:
+            dim = D_total // n_keys
+            return {k: dim for k in modality_keys}
+        else:
+            raise ValueError(
+                f"Cannot infer per-key dims for modality={modality!r}: "
+                f"D_total={D_total} is not evenly divisible by n_keys={n_keys}. "
+                f"Add `{attr} = {{key: dim, ...}}` to the DataConfig "
+                f"(keys={list(modality_keys)})."
+            )
+    return {k: 1 for k in modality_keys}
+
+
+#: Width of each rotation representation, used to size a key after
+#: ``StateActionTransform`` has converted it (see ``target_rotations``).
+ROTATION_DIMS: Dict[str, int] = {
+    "axis_angle": 3,
+    "euler_angles": 3,
+    "quaternion": 4,
+    "rotation_6d": 6,
+    "matrix": 9,
+}
+
+
+def _rotation_dim(rotation_type: str) -> int:
+    """Width of ``rotation_type``; ``euler_angles_*`` carries a convention suffix."""
+    if rotation_type.startswith("euler_angles"):
+        return ROTATION_DIMS["euler_angles"]
+    if rotation_type not in ROTATION_DIMS:
+        raise ValueError(
+            f"Unknown rotation representation {rotation_type!r}; "
+            f"expected one of {sorted(ROTATION_DIMS)}"
+        )
+    return ROTATION_DIMS[rotation_type]
+
+
+def _target_rotations(transform: ComposedModalityTransform) -> Dict[str, str]:
+    """Collect ``key -> target rotation`` from every StateActionTransform."""
+    targets: Dict[str, str] = {}
+    for sub_transform in getattr(transform, "transforms", []):
+        for key, rotation in getattr(sub_transform, "target_rotations", {}).items():
+            targets[key] = rotation
+    return targets
+
+
+def _model_space_dims(
+    raw_key_dims: Dict[str, int],
+    target_rotations: Dict[str, str],
+) -> Dict[str, int]:
+    """Per-key widths *after* the training transforms.
+
+    The stats file and the robot speak raw widths (a quaternion is 4); the model
+    tensor is what ``target_rotations`` produced (rotation_6d is 6). For
+    xtrainer2 this is the 16 -> 20 expansion.
+    """
+    return {
+        key: _rotation_dim(target_rotations[key]) if key in target_rotations else dim
+        for key, dim in raw_key_dims.items()
+    }
+
+
+def _build_dataset_metadata(
+    stats_for_key: Dict[str, Any],
+    embodiment_tag: Any,
+    action_keys: Sequence[str],
+    state_keys: Sequence[str],
+    action_key_dims: Optional[Dict[str, int]] = None,
+    state_key_dims: Optional[Dict[str, int]] = None,
+    rotation_types: Optional[Dict[str, str]] = None,
+) -> DatasetMetadata:
+    """Convert the *combined* stats arrays from ``dataset_statistics.json``
+    back into a per-subkey :class:`DatasetMetadata` matching what the training
+    pipeline produced.
+
+    The saved ``dataset_statistics.json`` stores stats as flat arrays of
+    length ``D = sum(per_key_dims)``.  For each ``"action.<sub>"`` key we
+    slice out ``dim_k`` elements starting at the current cursor and store them
+    under ``statistics.action.<sub> = {"min": [v0..v_{k-1}], ...}``.
+
+    Args:
+        stats_for_key: ``norm_stats[unnorm_key]`` from ``dataset_statistics.json``.
+        embodiment_tag: Embodiment tag from the DataConfig.
+        action_keys: Ordered list of full action keys.
+        state_keys: Ordered list of full state keys.
+        action_key_dims: Per-key dimension dict (``{full_key: dim_k}``).
+            Defaults to dim=1 for every key.
+        state_key_dims: Per-key dimension dict for state keys.
+            Defaults to dim=1 for every key.
+        rotation_types: ``full_key -> source rotation representation`` for keys
+            the training transforms rotate. Without it those transforms cannot
+            be rebuilt, because the dataset's ``modality.json`` is not shipped
+            with the checkpoint.
+    """
+    if action_key_dims is None:
+        action_key_dims = {k: 1 for k in action_keys}
+    if state_key_dims is None:
+        state_key_dims = {k: 1 for k in state_keys}
+    rotation_types = rotation_types or {}
+
+    def _split_combined(
+        combined: Dict[str, Sequence[float]],
+        keys: Sequence[str],
+        key_dims: Dict[str, int],
+    ):
+        """Split combined arrays into per-subkey dicts using per-key dims.
+
+        ``combined`` looks like ``{"min": [..D..], "max": [..D..], "mask": [..D..], ...}``.
+        ``keys`` is the ordered list of full keys.
+        ``key_dims`` maps each full key to its integer dimension.
+        Returns ``(stats_per_subkey, meta_per_subkey)``.
+        """
+        stats_per_subkey: Dict[str, Dict[str, List[float]]] = {}
+        meta_per_subkey: Dict[str, StateActionMetadata] = {}
+        cursor = 0
+        for full_key in keys:
+            subkey = full_key.split(".", 1)[1]
+            dim_k = key_dims.get(full_key, 1)
+            per_key: Dict[str, List[float]] = {}
+            for stat_name, arr in combined.items():
+                if stat_name == "mask":
+                    continue
+                end = cursor + dim_k
+                if end > len(arr):
+                    # Saved combined array shorter than expected (truncated
+                    # pad channels etc.). Skip this stat field silently.
+                    continue
+                per_key[stat_name] = [float(v) for v in arr[cursor:end]]
+            stats_per_subkey[subkey] = per_key
+            meta_per_subkey[subkey] = StateActionMetadata(
+                absolute=True,
+                rotation_type=rotation_types.get(full_key),
+                shape=(dim_k,),
+                continuous=True,
+            )
+            cursor += dim_k
+        return stats_per_subkey, meta_per_subkey
+
+    action_combined = stats_for_key.get("action", {})
+    state_combined = stats_for_key.get("state", {})
+
+    action_stats, action_meta = _split_combined(action_combined, action_keys, action_key_dims)
+    state_stats, state_meta = _split_combined(state_combined, state_keys, state_key_dims)
+
+    # Pydantic accepts dict input with field validators
+    return DatasetMetadata.model_validate(
+        {
+            "statistics": {
+                "state": state_stats,
+                "action": action_stats,
+            },
+            "modalities": {
+                "video": {},
+                "state": state_meta,
+                "action": action_meta,
+            },
+            "embodiment_tag": embodiment_tag.value
+            if hasattr(embodiment_tag, "value")
+            else embodiment_tag,
+        }
+    )
+
+
+class PolicyNormProcessor:
+    """Server-side normalization helper backed by training-time transforms.
+
+    Construct once per checkpoint; call :meth:`unapply_actions` to convert
+    a normalized action chunk ``(T, D)`` back to env-space.
+
+    Args:
+        ckpt_path: Path to the ``*.pt`` checkpoint (the loader looks for
+            ``config.yaml`` and ``dataset_statistics.json`` two dirs up).
+        unnorm_key: Which top-level key in ``dataset_statistics.json`` to use.
+            ``None`` → auto-pick the only key.
+    """
+
+    def __init__(self, ckpt_path: str, unnorm_key: Optional[str] = None) -> None:
+        self._ckpt_path = str(ckpt_path)
+        cfg, norm_stats = read_mode_config(self._ckpt_path)
+        self._model_cfg = cfg
+        self._norm_stats = norm_stats
+
+        # 3-early) Pick the requested unnorm_key (or auto-select) BEFORE
+        # resolving robot_type so we can use it as a hint for multi-robot mixtures.
+        if unnorm_key is None:
+            if len(norm_stats) == 1:
+                unnorm_key = next(iter(norm_stats.keys()))
+            # else: defer error to step 3 below after robot_type resolution attempt
+        elif unnorm_key not in norm_stats:
+            raise KeyError(
+                f"unnorm_key={unnorm_key!r} not in {list(norm_stats.keys())}"
+            )
+        self._unnorm_key = unnorm_key  # may still be None for multi-key case
+
+        # 1) Resolve which DataConfig was used at training.
+        robot_type = _resolve_robot_type(cfg, unnorm_key=unnorm_key)
+        if robot_type not in ROBOT_TYPE_CONFIG_MAP:
+            raise KeyError(
+                f"robot_type={robot_type!r} not in ROBOT_TYPE_CONFIG_MAP "
+                f"(available: {sorted(ROBOT_TYPE_CONFIG_MAP.keys())}). "
+                "Make sure the example's data_registry/data_config.py is importable."
+            )
+        self._data_config = ROBOT_TYPE_CONFIG_MAP[robot_type]
+        self._action_keys: List[str] = list(self._data_config.action_keys)
+        self._state_keys: List[str] = list(getattr(self._data_config, "state_keys", []))
+        self._modality_key_set = set(self._action_keys) | set(self._state_keys)
+
+        # 2) Build training-time transform pipeline.
+        transform = self._data_config.transform()
+        if not isinstance(transform, ComposedModalityTransform):
+            transform = ComposedModalityTransform(transforms=[transform])
+        self._transform = transform
+
+        # 3) Pick the requested unnorm_key (finalize; error if still None here).
+        if self._unnorm_key is None:
+            raise ValueError(
+                f"Multiple unnorm_keys in dataset_statistics.json: "
+                f"{list(norm_stats.keys())}. Pass unnorm_key explicitly."
+            )
+        unnorm_key = self._unnorm_key
+
+        # 4) Resolve per-key dims (handles multi-d action/state keys).
+        stats_for_unnorm = norm_stats[unnorm_key]
+        self._action_key_dims: Dict[str, int] = _infer_key_dims(
+            self._data_config, stats_for_unnorm, self._action_keys, "action"
+        )
+        self._state_key_dims: Dict[str, int] = _infer_key_dims(
+            self._data_config, stats_for_unnorm, self._state_keys, "state"
+        )
+
+        # 5) Widths of the model tensor, which differ from the raw widths for
+        #    every key the transforms rotate (xtrainer2: quaternion 4 -> 6D).
+        target_rotations = _target_rotations(self._transform)
+        self._action_model_dims = _model_space_dims(self._action_key_dims, target_rotations)
+        self._state_model_dims = _model_space_dims(self._state_key_dims, target_rotations)
+
+        # 6) Build & bind metadata.
+        ds_meta = _build_dataset_metadata(
+            stats_for_key=stats_for_unnorm,
+            embodiment_tag=self._data_config.embodiment_tag,
+            action_keys=self._action_keys,
+            state_keys=self._state_keys,
+            action_key_dims=self._action_key_dims,
+            state_key_dims=self._state_key_dims,
+            rotation_types=dict(getattr(self._data_config, "rotation_types", {})),
+        )
+        # Only the state/action transforms are bound. The video transforms would
+        # demand per-camera resolutions that the checkpoint does not carry, and
+        # the server never runs them: images are resized by the framework's own
+        # inference path.
+        self._state_action_transforms = [
+            sub_transform
+            for sub_transform in self._transform.transforms
+            if self._modality_key_set.intersection(getattr(sub_transform, "apply_to", []))
+        ]
+        for sub_transform in self._state_action_transforms:
+            sub_transform.set_metadata(ds_meta)
+            sub_transform.eval()  # mark transforms as eval-mode
+
+        logger.info(
+            "PolicyNormProcessor ready: robot_type=%s, unnorm_key=%s, "
+            "action_keys=%s (raw dims=%s, model dims=%s), state_keys=%s "
+            "(raw dims=%s, model dims=%s)",
+            robot_type,
+            unnorm_key,
+            self._action_keys,
+            [self._action_key_dims[k] for k in self._action_keys],
+            [self._action_model_dims[k] for k in self._action_keys],
+            self._state_keys,
+            [self._state_key_dims[k] for k in self._state_keys],
+            [self._state_model_dims[k] for k in self._state_keys],
+        )
+
+    # ------------------------------------------------------------------
+    # Public properties
+    # ------------------------------------------------------------------
+    @property
+    def action_keys(self) -> List[str]:
+        return list(self._action_keys)
+
+    @property
+    def state_keys(self) -> List[str]:
+        return list(self._state_keys)
+
+    @property
+    def unnorm_key(self) -> str:
+        return self._unnorm_key
+
+    @property
+    def available_unnorm_keys(self) -> List[str]:
+        return list(self._norm_stats.keys())
+
+    @property
+    def transform(self) -> ComposedModalityTransform:
+        return self._transform
+
+    @property
+    def action_dim(self) -> int:
+        """Width of the model action vector (post-transform)."""
+        return sum(self._action_model_dims[key] for key in self._action_keys)
+
+    @property
+    def raw_action_dim(self) -> int:
+        """Width of an action in robot/dataset coordinates."""
+        return sum(self._action_key_dims[key] for key in self._action_keys)
+
+    @property
+    def state_dim(self) -> int:
+        """Width of the model state vector (post-transform)."""
+        return sum(self._state_model_dims[key] for key in self._state_keys)
+
+    @property
+    def raw_state_dim(self) -> int:
+        """Width of a state as the robot reports it."""
+        return sum(self._state_key_dims[key] for key in self._state_keys)
+
+    # ------------------------------------------------------------------
+    # Forward path (robot observation → model state)
+    # ------------------------------------------------------------------
+    def apply_state(self, raw_state: np.ndarray) -> np.ndarray:
+        """Normalize a raw state exactly as the training dataloader did.
+
+        Args:
+            raw_state: ``(raw_state_dim,)`` or ``(T, raw_state_dim)`` in dataset
+                coordinates -- i.e. after any robot-specific reordering, with
+                rotations still in their source representation.
+
+        Returns:
+            ``(T, state_dim)`` normalized state, the layout the action head was
+            trained on.
+        """
+        state = np.asarray(raw_state, dtype=np.float32)
+        if state.ndim == 1:
+            state = state[None, :]
+        if state.ndim != 2:
+            raise ValueError(f"Expected (raw_state_dim,) or (T, raw_state_dim); got {state.shape}")
+        if state.shape[-1] != self.raw_state_dim:
+            raise ValueError(
+                f"Expected {self.raw_state_dim} raw state dims "
+                f"(state_keys={self._state_keys}, dims="
+                f"{[self._state_key_dims[k] for k in self._state_keys]}); got {state.shape[-1]}"
+            )
+
+        data: Dict[str, Any] = {}
+        cursor = 0
+        for full_key in self._state_keys:
+            dim_k = self._state_key_dims[full_key]
+            data[full_key] = state[:, cursor : cursor + dim_k]
+            cursor += dim_k
+
+        for sub_transform in self._state_action_transforms:
+            data = sub_transform(data)
+
+        parts: List[np.ndarray] = []
+        for full_key in self._state_keys:
+            value = data[full_key]
+            if isinstance(value, torch.Tensor):
+                value = value.detach().cpu().numpy()
+            parts.append(np.asarray(value, dtype=np.float32))
+        return np.concatenate(parts, axis=-1)
+
+    # ------------------------------------------------------------------
+    # Inverse path (model output → env action)
+    # ------------------------------------------------------------------
+    def unapply_actions(self, normalized_actions: np.ndarray) -> np.ndarray:
+        """Invert action normalization using the training-time pipeline.
+
+        Args:
+            normalized_actions: shape ``(T, D)`` where ``D == action_dim``.
+
+        Returns:
+            ``(T, raw_action_dim)`` un-normalized actions in env coordinates.
+        """
+        normalized_actions = np.asarray(normalized_actions)
+        assert normalized_actions.ndim == 2, (
+            f"Expected (T, D); got shape {normalized_actions.shape}"
+        )
+
+        # Split (T, D) into per-key {full_key: torch.Tensor[T, dim_k]}.
+        data: Dict[str, torch.Tensor] = {}
+        cursor = 0
+        for full_key in self._action_keys:
+            dim_k = self._action_model_dims.get(full_key, 1)
+            slice_ = normalized_actions[..., cursor : cursor + dim_k]
+            data[full_key] = torch.as_tensor(slice_, dtype=torch.float32)
+            cursor += dim_k
+
+        if cursor != normalized_actions.shape[-1]:
+            raise ValueError(
+                f"Sum of per-key model dims ({cursor}) != action_dim "
+                f"({normalized_actions.shape[-1]}). "
+                f"action_keys={self._action_keys}, "
+                f"action_model_dims={self._action_model_dims}"
+            )
+
+        out: Dict[str, Any] = data
+        for sub_transform in reversed(self._state_action_transforms):
+            out = sub_transform.unapply(out)
+
+        parts: List[np.ndarray] = []
+        for full_key in self._action_keys:
+            v = out[full_key]
+            if isinstance(v, torch.Tensor):
+                v = v.detach().cpu().numpy()
+            parts.append(np.asarray(v))
+        return np.concatenate(parts, axis=-1)

real_robot/source/deployment/model_server/policy_wrapper.py

@@ -0,0 +1,280 @@
+# Copyright 2025 starVLA community. All rights reserved.
+# Licensed under the MIT License.
+"""Policy server wrapper.
+
+Encapsulates a `baseframework` instance plus a :class:`PolicyNormProcessor`
+that reuses the *training-time* :class:`ComposedModalityTransform` for action
+un-normalization (no hand-rolled math). The websocket server returns
+already-unnormalized actions.
+
+Client-side responsibilities that REMAIN on the client:
+  - environment-specific adapters (image_history, gripper sticky, action
+    ensembling)
+  - chunk-cache scheduling (`step % chunk_size == 0` triggers a new infer)
+
+Embodiments whose DataConfig declares a ``robot_command_layout`` (currently
+xtrainer2) additionally get the raw-robot conversions here, so the client never
+has to reproduce training-time normalization or the quaternion order:
+
+  - ``state`` arrives in robot coordinates (16-D, xyzw) and is reordered to the
+    training layout, then normalized with the training transforms (20-D)
+  - returned actions are un-normalized *and* reordered back to xyzw
+
+Exposed API:
+  - ``metadata`` (dict, sent at handshake): ``action_chunk_size``,
+    ``available_unnorm_keys``, ``action_keys``, ``state_keys``, plus the
+    embodiment's raw-robot contract.
+  - ``predict_action(examples, unnorm_key=None, **kwargs)`` returns
+    ``{"actions": np.ndarray[B, T, action_dim]}``.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any, Dict, List, Optional
+
+import numpy as np
+import torch
+
+from starVLA.model.framework.base_framework import baseframework
+from starVLA.model.framework.share_tools import read_mode_config
+
+from deployment.model_server.policy_norm_processor import PolicyNormProcessor
+from deployment.model_server import xtrainer2_command
+
+
+class PolicyServerWrapper:
+    """Wraps a `baseframework` for use as a websocket-server policy."""
+
+    def __init__(
+        self,
+        ckpt_path: str,
+        device: str = "cuda",
+        use_bf16: bool = False,
+        unnorm_key: Optional[str] = None,
+        normalized_action_clip: Optional[float] = None,
+    ) -> None:
+        self._ckpt_path = str(ckpt_path)
+        self._normalized_action_clip = normalized_action_clip
+
+        logging.info("PolicyServerWrapper: loading framework from %s", self._ckpt_path)
+        framework = baseframework.from_pretrained(self._ckpt_path)
+        if use_bf16:
+            framework = framework.to(torch.bfloat16)
+        framework = framework.to(device).eval()
+        self._framework = framework
+
+        # Co-located metadata.
+        model_cfg, _ = read_mode_config(self._ckpt_path)
+        self._model_cfg = model_cfg
+        vla_data_cfg = model_cfg.get("datasets", {}).get("vla_data", {})
+        include_state = vla_data_cfg.get("include_state", False)
+        self._uses_state = include_state not in (False, None, 0, "False", "false", "0")
+
+        # action_chunk_size = future_action_window_size + 1 (matches old client).
+        action_model_cfg = model_cfg["framework"]["action_model"]
+        
+        if "action_horizon" in action_model_cfg:
+            self._action_chunk_size = int(action_model_cfg["action_horizon"])
+        elif "future_action_window_size" in action_model_cfg:
+            self._action_chunk_size = int(action_model_cfg["future_action_window_size"]) + 1
+        else:
+            raise ValueError(
+                f"PolicyServerWrapper: no action_horizon or future_action_window_size found in model config for {self._ckpt_path}"
+            )
+        # Cache of PolicyNormProcessor instances per unnorm_key.
+        # For single-dataset ckpts unnorm_key is auto-selected; for multi-dataset
+        # ckpts clients must pass unnorm_key per request.
+        self._default_unnorm_key = unnorm_key
+        self._norm_processors: Dict[str, PolicyNormProcessor] = {}
+
+        # Peek at available keys without building a full processor.
+        _, _ns = read_mode_config(self._ckpt_path)
+        self._available_unnorm_keys: List[str] = list(_ns.keys())
+
+        # Eagerly build when unambiguous; defer for multi-key / no explicit key.
+        if unnorm_key is not None or len(self._available_unnorm_keys) == 1:
+            default_proc = self._get_processor(unnorm_key)
+            self._default_unnorm_key = default_proc.unnorm_key
+            logging.info(
+                "PolicyServerWrapper ready: action_chunk_size=%d, default_unnorm_key=%s, "
+                "available_unnorm_keys=%s, action_keys=%s, state_keys=%s",
+                self._action_chunk_size,
+                default_proc.unnorm_key,
+                default_proc.available_unnorm_keys,
+                default_proc.action_keys,
+                default_proc.state_keys,
+            )
+        else:
+            logging.info(
+                "PolicyServerWrapper ready (multi-key): action_chunk_size=%d, "
+                "available_unnorm_keys=%s — clients must pass unnorm_key per request.",
+                self._action_chunk_size,
+                self._available_unnorm_keys,
+            )
+
+    def _get_processor(self, unnorm_key: Optional[str]) -> PolicyNormProcessor:
+        cache_key = unnorm_key if unnorm_key is not None else "__default__"
+        if cache_key not in self._norm_processors:
+            self._norm_processors[cache_key] = PolicyNormProcessor(
+                self._ckpt_path, unnorm_key=unnorm_key
+            )
+        return self._norm_processors[cache_key]
+
+    @staticmethod
+    def _uses_xtrainer2_layout(proc: PolicyNormProcessor) -> bool:
+        """Whether this embodiment speaks the raw xtrainer2 robot layout."""
+        layout = getattr(proc._data_config, "robot_command_layout", None)
+        return layout == xtrainer2_command.COMMAND_LAYOUT
+
+    @property
+    def metadata(self) -> Dict[str, Any]:
+        """Model-invariant metadata; sent to client at websocket handshake."""
+        base = {
+            "env": "starvla_policy_server",
+            "ckpt_path": self._ckpt_path,
+            "action_chunk_size": self._action_chunk_size,
+            "uses_state": self._uses_state,
+            "available_unnorm_keys": self._available_unnorm_keys,
+            "default_unnorm_key": self._default_unnorm_key,
+            "normalized_action_clip": self._normalized_action_clip,
+            "action_head_attention_mask": False,
+            "inference_rng": "continuous_server_stream",
+            # Every client sends images this way; the framework does its own
+            # resize, so only the layout and channel order are binding.
+            "image_format": "HWC_RGB_uint8",
+            "image_key": "image",
+        }
+        # Enrich with per-embodiment keys when a default processor already exists.
+        if self._default_unnorm_key is not None:
+            proc = self._get_processor(self._default_unnorm_key)
+            base["action_keys"] = proc.action_keys
+            base["state_keys"] = proc.state_keys
+            base["camera_order"] = list(getattr(proc._data_config, "video_keys", []))
+            base["model_action_dim"] = proc.action_dim
+            base["model_state_dim"] = proc.state_dim
+            base["server_normalizes_state"] = self._server_normalizes_state(proc)
+            if base["server_normalizes_state"]:
+                base["state_dim"] = proc.raw_state_dim
+            if self._uses_xtrainer2_layout(proc):
+                base["robot_command_layout"] = xtrainer2_command.COMMAND_LAYOUT
+                base["action_dim"] = proc.raw_action_dim
+                base["model_quaternion_format"] = xtrainer2_command.MODEL_QUATERNION_FORMAT
+                base["robot_quaternion_format"] = xtrainer2_command.SOURCE_QUATERNION_FORMAT
+                base["quaternion_starts"] = list(xtrainer2_command.QUATERNION_STARTS)
+                action_mode = str(
+                    getattr(self, "_model_cfg", {})
+                    .get("datasets", {})
+                    .get("vla_data", {})
+                    .get("action_mode", "abs")
+                )
+                base["action_mode"] = action_mode
+                base["server_composes_relative_eef"] = (
+                    xtrainer2_command.is_relative_action_mode(action_mode)
+                )
+        return base
+
+    @staticmethod
+    def _server_normalizes_state(proc: PolicyNormProcessor) -> bool:
+        """Whether ``state`` arrives raw and must be normalized server-side."""
+        return bool(getattr(proc._data_config, "server_normalizes_state", False))
+
+    @classmethod
+    def _prepare_examples(cls, examples: List[dict], proc: PolicyNormProcessor) -> List[dict]:
+        """Turn raw-robot observations into the tensors the model was trained on.
+
+        Leaves examples untouched for embodiments that do not opt in, so clients
+        that already send normalized state keep working.
+        """
+        if not cls._server_normalizes_state(proc):
+            return examples
+
+        to_model_layout = (
+            xtrainer2_command.robot_state_to_model
+            if cls._uses_xtrainer2_layout(proc)
+            else None
+        )
+
+        prepared: List[dict] = []
+        for example in examples:
+            if "state" not in example or example["state"] is None:
+                prepared.append(example)
+                continue
+            raw_state = np.asarray(example["state"], dtype=np.float32)
+            if raw_state.shape[-1] != proc.raw_state_dim:
+                raise ValueError(
+                    f"state must be the robot's raw {proc.raw_state_dim}-D vector "
+                    f"(state_keys={proc.state_keys}); got shape {raw_state.shape}. "
+                    "The server applies the training-time normalization itself."
+                )
+            if to_model_layout is not None:
+                raw_state = to_model_layout(raw_state)
+            example = dict(example)
+            # Keep the training-layout (wxyz) pose so rel-mode actions can be
+            # composed back to absolute commands after un-normalization.
+            example["_raw_state_wxyz"] = raw_state
+            example["state"] = proc.apply_state(raw_state)
+            prepared.append(example)
+        return prepared
+
+    def predict_action(
+        self,
+        examples: List[dict],
+        unnorm_key: Optional[str] = None,
+        **kwargs,
+    ) -> Dict[str, np.ndarray]:
+        """Run the framework, then un-normalize via training-time transforms.
+
+        Args:
+            examples: list of dicts (each with ``image`` / ``lang`` / optional
+                ``state``). For embodiments that declare
+                ``server_normalizes_state``, ``state`` is the robot's raw vector
+                and is normalized here.
+            unnorm_key: dataset key for un-normalization stats. ``None`` -->
+                use the wrapper's default (auto-picked at startup).
+            **kwargs: forwarded to the framework's ``predict_action``
+                (``do_sample``, ``use_ddim``, ``num_ddim_steps``, ...).
+
+        Returns:
+            ``{"actions": np.ndarray[B, T, D]}`` -- un-normalized.
+        """
+        effective_key = unnorm_key if unnorm_key is not None else self._default_unnorm_key
+        if effective_key is None:
+            if len(self._available_unnorm_keys) == 1:
+                effective_key = self._available_unnorm_keys[0]
+            else:
+                raise ValueError(
+                    f"predict_action: unnorm_key not specified and no default set. "
+                    f"Pass one of {self._available_unnorm_keys}."
+                )
+        proc = self._get_processor(effective_key)
+        examples = self._prepare_examples(examples, proc)
+
+        out = self._framework.predict_action(examples=examples, **kwargs)
+        normalized = np.asarray(out["normalized_actions"])  # (B, T, D)
+        if self._normalized_action_clip is not None:
+            limit = float(self._normalized_action_clip)
+            normalized = np.clip(normalized, -limit, limit)
+
+        unnorm = np.stack(
+            [proc.unapply_actions(normalized[b]) for b in range(normalized.shape[0])],
+            axis=0,
+        )
+        if self._uses_xtrainer2_layout(proc):
+            if xtrainer2_command.is_relative_action_mode(
+                self._model_cfg.get("datasets", {}).get("vla_data", {}).get("action_mode")
+            ):
+                composed = []
+                for batch_index in range(unnorm.shape[0]):
+                    state_wxyz = examples[batch_index].get("_raw_state_wxyz")
+                    if state_wxyz is None:
+                        raise ValueError(
+                            "action_mode=rel requires a raw robot state so relative "
+                            "EE commands can be composed back to absolute poses"
+                        )
+                    composed.append(
+                        xtrainer2_command.compose_relative_eef_wxyz(unnorm[batch_index], state_wxyz)
+                    )
+                unnorm = np.stack(composed, axis=0)
+            unnorm = xtrainer2_command.model_actions_to_robot(unnorm)
+        return {"actions": unnorm}

real_robot/source/deployment/model_server/server_policy.py

@@ -0,0 +1,79 @@
+# Copyright 2025 starVLA community. All rights reserved.
+# Licensed under the MIT License, Version 1.0 (the "License"); 
+
+import logging
+import socket
+import argparse
+from deployment.model_server.tools.websocket_policy_server import WebsocketPolicyServer
+from deployment.model_server.policy_wrapper import PolicyServerWrapper
+import os
+
+
+def main(args) -> None:
+    # Example usage:
+    # policy = YourPolicyClass()  # Replace with your actual policy class
+    # server = WebsocketPolicyServer(policy, host="localhost", port=10091)
+    # server.serve_forever()
+
+    policy = PolicyServerWrapper(
+        ckpt_path=args.ckpt_path,
+        device="cuda",
+        use_bf16=args.use_bf16,
+        unnorm_key=args.unnorm_key,
+    )
+
+    hostname = socket.gethostname()
+    local_ip = socket.gethostbyname(hostname)
+    logging.info("Creating server (host: %s, ip: %s)", hostname, local_ip)
+    # The handshake metadata is the request contract; log it so a client
+    # mismatch (camera order, state width, quaternion order) is visible here.
+    for key, value in sorted(policy.metadata.items()):
+        logging.info("  metadata %-24s %s", key, value)
+
+    # start websocket server
+    server = WebsocketPolicyServer(
+        policy=policy,
+        host="0.0.0.0",
+        port=args.port,
+        idle_timeout=args.idle_timeout,
+        metadata={**policy.metadata, "policy_seed": args.seed},
+    )
+    logging.info("server running ...")
+    server.serve_forever()
+
+
+def build_argparser():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--ckpt_path", type=str, default="Qwen/Qwen2.5-VL-3B-Instruct")
+    parser.add_argument("--port", type=int, default=10093)
+    parser.add_argument("--use_bf16", action="store_true")
+    parser.add_argument(
+        "--unnorm_key",
+        type=str,
+        default=None,
+        help="dataset_statistics.json key for un-normalization; required only for multi-dataset checkpoints",
+    )
+    parser.add_argument("--seed", type=int, default=0)
+    parser.add_argument("--idle_timeout" , type=int, default=1800, help="Idle timeout in seconds, -1 means never close")
+    return parser
+
+
+def start_debugpy_once():
+    """start debugpy once"""
+    import debugpy
+    if getattr(start_debugpy_once, "_started", False):
+        return
+    debugpy.listen(("0.0.0.0", 10095))
+    print("Waiting for VSCode attach on 0.0.0.0:10095 ...")
+    debugpy.wait_for_client()
+    start_debugpy_once._started = True
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO, force=True)
+    parser = build_argparser()
+    args = parser.parse_args()
+    if os.getenv("DEBUG", "").strip().lower() in {"1", "true", "yes"}:
+        print("DEBUGPY is enabled")
+        start_debugpy_once()
+    main(args)

real_robot/source/deployment/model_server/tools/image_tools.py

@@ -0,0 +1,123 @@
+import numpy as np
+from PIL import Image
+
+
+def convert_to_uint8(img: np.ndarray) -> np.ndarray:
+    """Converts an image to uint8 if it is a float image.
+
+    This is important for reducing the size of the image when sending it over the network.
+    """
+    if np.issubdtype(img.dtype, np.floating):
+        img = (255 * img).astype(np.uint8)
+    return img
+
+
+def resize_with_pad(images: np.ndarray, height: int, width: int, method=Image.BILINEAR) -> np.ndarray:
+    """Replicates tf.image.resize_with_pad for multiple images using PIL. Resizes a batch of images to a target height.
+
+    Args:
+        images: A batch of images in [..., height, width, channel] format.
+        height: The target height of the image.
+        width: The target width of the image.
+        method: The interpolation method to use. Default is bilinear.
+
+    Returns:
+        The resized images in [..., height, width, channel].
+    """
+    # If the images are already the correct size, return them as is.
+    if images.shape[-3:-1] == (height, width):
+        return images
+
+    original_shape = images.shape
+
+    images = images.reshape(-1, *original_shape[-3:])
+    resized = np.stack([_resize_with_pad_pil(Image.fromarray(im), height, width, method=method) for im in images])
+    return resized.reshape(*original_shape[:-3], *resized.shape[-3:])
+
+
+def _resize_with_pad_pil(image: Image.Image, height: int, width: int, method: int) -> Image.Image:
+    """Replicates tf.image.resize_with_pad for one image using PIL. Resizes an image to a target height and
+    width without distortion by padding with zeros.
+
+    Unlike the jax version, note that PIL uses [width, height, channel] ordering instead of [batch, h, w, c].
+    """
+    cur_width, cur_height = image.size
+    if cur_width == width and cur_height == height:
+        return image  # No need to resize if the image is already the correct size.
+
+    ratio = max(cur_width / width, cur_height / height)
+    resized_height = int(cur_height / ratio)
+    resized_width = int(cur_width / ratio)
+    resized_image = image.resize((resized_width, resized_height), resample=method)
+
+    zero_image = Image.new(resized_image.mode, (width, height), 0)
+    pad_height = max(0, int((height - resized_height) / 2))
+    pad_width = max(0, int((width - resized_width) / 2))
+    zero_image.paste(resized_image, (pad_width, pad_height))
+    assert zero_image.size == (width, height)
+    return zero_image
+
+
+from typing import Any
+
+
+def to_pil_preserve(images: Any, scale_float: bool = True):
+    """
+    Convert (possibly nested) numpy image arrays back to PIL.Image WITHOUT changing spatial shape
+    or nesting structure.
+
+    Accepts:
+      - np.ndarray with shape (H, W, C), C in {1,3,4}, dtype uint8 or float
+      - PIL.Image.Image (returned as-is)
+      - Nested list / tuple structures containing the above
+
+    Guarantees:
+      - No resize / pad / crop performed
+      - Returns an object with the SAME nesting layout (list -> list, tuple -> tuple)
+      - Only dtype (float -> uint8) and channel-mode adaptation may happen
+        * float arrays assumed in [0,1] if scale_float=True (scaled *255 + clip)
+    Args:
+      images: input object / sequence
+      scale_float: whether to scale float images in [0,1] to uint8
+    Returns:
+      Mirrored structure with all leaf nodes as PIL.Image.Image
+    """
+
+    def _convert(obj):
+        # Nested containers
+        if isinstance(obj, list):
+            return [_convert(x) for x in obj]
+        if isinstance(obj, tuple):
+            return tuple(_convert(x) for x in obj)
+
+        # PIL stays
+        if isinstance(obj, Image.Image):
+            return obj
+
+        # numpy -> PIL
+        if isinstance(obj, np.ndarray):
+            arr = obj
+            if arr.ndim != 3:
+                raise ValueError(f"Expected 3D array (H,W,C), got shape={arr.shape}")
+            if arr.shape[2] not in (1, 3, 4):
+                raise ValueError(f"Channel count must be 1/3/4, got {arr.shape[2]}")
+            if np.issubdtype(arr.dtype, np.floating):
+                if scale_float:
+                    arr = np.clip(arr, 0.0, 1.0)
+                    arr = (arr * 255.0 + 0.5).astype(np.uint8)
+                else:
+                    raise TypeError("Float array provided but scale_float=False")
+            elif arr.dtype != np.uint8:
+                arr = arr.astype(np.uint8)
+
+            # Single channel -> 'L'
+            if arr.shape[2] == 1:
+                arr = arr[:, :, 0]
+                return Image.fromarray(arr, mode="L")
+            # 3 channels -> RGB, 4 -> RGBA
+            mode = "RGB" if arr.shape[2] == 3 else "RGBA"
+            return Image.fromarray(arr, mode=mode)
+
+        raise TypeError(f"Unsupported element type: {type(obj)}")
+
+    return _convert(images)

real_robot/source/deployment/model_server/tools/msgpack_numpy.py

@@ -0,0 +1,59 @@
+"""Adds NumPy array support to msgpack.
+
+msgpack is good for (de)serializing data over a network for multiple reasons:
+- msgpack is secure (as opposed to pickle/dill/etc which allow for arbitrary code execution)
+- msgpack is widely used and has good cross-language support
+- msgpack does not require a schema (as opposed to protobuf/flatbuffers/etc) which is convenient in dynamically typed
+    languages like Python and JavaScript
+- msgpack is fast and efficient (as opposed to readable formats like JSON/YAML/etc); I found that msgpack was ~4x faster
+    than pickle for serializing large arrays using the below strategy
+
+The code below is adapted from https://github.com/lebedov/msgpack-numpy.
+Its original BSD license is retained in licenses/msgpack-numpy-LICENSE.
+The reason not to use that library directly is
+that it falls back to pickle for object arrays.
+"""
+
+import functools
+
+import msgpack
+import numpy as np
+
+
+def pack_array(obj):
+    if (isinstance(obj, (np.ndarray, np.generic))) and obj.dtype.kind in ("V", "O", "c"):
+        raise ValueError(f"Unsupported dtype: {obj.dtype}")
+
+    if isinstance(obj, np.ndarray):
+        return {
+            b"__ndarray__": True,
+            b"data": obj.tobytes(),
+            b"dtype": obj.dtype.str,
+            b"shape": obj.shape,
+        }
+
+    if isinstance(obj, np.generic):
+        return {
+            b"__npgeneric__": True,
+            b"data": obj.item(),
+            b"dtype": obj.dtype.str,
+        }
+
+    return obj
+
+
+def unpack_array(obj):
+    if b"__ndarray__" in obj:
+        return np.ndarray(buffer=obj[b"data"], dtype=np.dtype(obj[b"dtype"]), shape=obj[b"shape"])
+
+    if b"__npgeneric__" in obj:
+        return np.dtype(obj[b"dtype"]).type(obj[b"data"])
+
+    return obj
+
+
+Packer = functools.partial(msgpack.Packer, default=pack_array)
+packb = functools.partial(msgpack.packb, default=pack_array)
+
+Unpacker = functools.partial(msgpack.Unpacker, object_hook=unpack_array)
+unpackb = functools.partial(msgpack.unpackb, object_hook=unpack_array)

real_robot/source/deployment/model_server/tools/websocket_policy_client.py

@@ -0,0 +1,75 @@
+# Copyright 2025 starVLA community. All rights reserved.
+# Licensed under the MIT License, Version 1.0 (the "License"); 
+
+import logging, argparse
+import time, os
+from typing import Dict, Optional, Tuple
+
+from typing_extensions import override
+import websockets.sync.client
+
+from . import msgpack_numpy
+
+
+class WebsocketClientPolicy:
+    """Implements the Policy interface by communicating with a server over websocket.
+
+    See WebsocketPolicyServer for a corresponding server implementation.
+    """
+
+    def __init__(self, host: str = "127.0.0.1", port: Optional[int] = 10093, api_key: Optional[str] = None) -> None:
+        # 0.0.0.0 cannot be used as a connection target, here default 127.0.0.1
+        self._uri = f"ws://{host}"
+        if port is not None:
+            self._uri += f":{port}"
+        self._packer = msgpack_numpy.Packer()
+        self._api_key = api_key
+        self._ws, self._server_metadata = self._wait_for_server()
+
+    def get_server_metadata(self) -> Dict:
+        return self._server_metadata
+
+    def _wait_for_server(self, timeout: float = 300) -> Tuple[websockets.sync.client.ClientConnection, Dict]:
+        logging.info(f"Waiting for server at {self._uri}...")
+        start_time = time.time()
+        
+        for k in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy", "ALL_PROXY", "all_proxy"):
+            os.environ.pop(k, None)
+        
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Failed to connect to server within {timeout} seconds")
+            
+            try:
+                headers = {"Authorization": f"Api-Key {self._api_key}"} if self._api_key else None
+                conn = websockets.sync.client.connect(
+                    self._uri,
+                    compression=None,
+                    max_size=None,
+                    additional_headers=headers,
+                    open_timeout=150,
+                    ping_interval=None,
+                    ping_timeout=None,
+                )
+                metadata = msgpack_numpy.unpackb(conn.recv())
+                return conn, metadata
+            except ConnectionRefusedError:
+                logging.info(f"Still waiting for server {self._uri} ...")
+                time.sleep(2)
+
+    def close(self) -> None:
+        try:
+            self._ws.close()
+        except Exception:
+            pass
+    
+    @override
+    def predict_action(self, query_info: Dict) -> Dict:
+        data = self._packer.pack(query_info)
+        self._ws.send(data)
+        response = self._ws.recv()
+        if isinstance(response, str):
+            raise RuntimeError(f"Error in inference server:\n{response}")
+        return msgpack_numpy.unpackb(response)
+
+

real_robot/source/deployment/model_server/tools/websocket_policy_server.py

@@ -0,0 +1,160 @@
+# Copyright 2025 starVLA community. All rights reserved.
+# Licensed under the MIT License, Version 1.0 (the "License"); 
+
+import asyncio
+import logging
+import traceback
+import time
+
+import websockets.asyncio.server
+import websockets.frames
+
+# from openpi_client import base_policy as _base_policy
+from . import msgpack_numpy
+
+class WebsocketPolicyServer:
+    """Serves a policy using the websocket protocol. See websocket_client_policy.py for a client implementation.
+
+    Currently only implements the `load` and `infer` methods.
+    """
+
+    def __init__(
+        self,
+        policy,
+        host: str = "0.0.0.0",
+        port: int = 10093,
+        idle_timeout: int = -1,  # Negative values disable the idle timeout.
+        metadata: dict | None = None,
+        
+    ) -> None:
+        self._policy = policy
+        self._host = host
+        self._port = port
+        self._metadata = metadata or {}
+        self._idle_timeout = idle_timeout
+        self._last_active = time.time()
+        logging.getLogger("websockets.server").setLevel(logging.INFO)
+
+    def serve_forever(self) -> None:
+        asyncio.run(self.run())
+
+    async def run(self):
+        async with websockets.asyncio.server.serve(
+            self._handler,
+            self._host,
+            self._port,
+            compression=None,
+            max_size=None,
+        ) as server:
+            if self._idle_timeout > 0:
+                await self._idle_watchdog(server)
+            else:
+                await server.serve_forever()
+
+    async def _idle_watchdog(self, server):
+        """Close the server after the configured idle period."""
+        while True:
+            await asyncio.sleep(5)
+            if time.time() - self._last_active > self._idle_timeout:
+                logging.info(f"Idle timeout ({self._idle_timeout}s) reached, shutting down server.")
+                server.close()
+                await server.wait_closed()
+                break
+
+    async def _handler(self, websocket: websockets.asyncio.server.ServerConnection):
+        logging.info(f"Connection from {websocket.remote_address} opened")
+        packer = msgpack_numpy.Packer()
+
+        await websocket.send(packer.pack(self._metadata))
+
+        while True:
+            try:
+                msg = msgpack_numpy.unpackb(await websocket.recv())
+                self._last_active = time.time()
+                ret = self._route_message(msg)
+                await websocket.send(packer.pack(ret))
+            except websockets.ConnectionClosed:
+                logging.info(f"Connection from {websocket.remote_address} closed")
+                break
+            except Exception:
+                await websocket.send(traceback.format_exc())
+                await websocket.close(
+                    code=websockets.frames.CloseCode.INTERNAL_ERROR,
+                    reason="Internal server error. Traceback included in previous frame.",
+                )
+                raise
+
+    # route logic: recognize request from client
+    def _route_message(self, msg: dict) -> dict:
+        """
+        Route rules (fault-tolerant):
+        - Supports messages of form:
+            {"type": "ping|init|infer|reset", "request_id": "...", "payload": {...}}
+          or a flat dict (will be treated as payload).
+        - Does NOT raise inside this function: all exceptions are caught and encoded in response.
+        """
+        if not isinstance(msg, dict):
+            return {
+                "status": "error",
+                "ok": False,
+                "type": "inference_result",
+                "request_id": "default",
+                "error": {"message": "Payload must be a dict", "payload_type": str(type(msg))},
+            }
+
+        req_id = msg.get("request_id", "default")
+        mtype = msg.get("type", "infer")          # default = infer
+        # The envelope fields are addressed to this router, not to the policy.
+        payload = {key: value for key, value in msg.items() if key not in ("type", "request_id")}
+
+        # ping
+        if mtype == "ping":
+            return {"status": "ok", "ok": True, "type": "ping", "request_id": req_id}
+
+        # infer --> framework.predict_action
+        elif mtype == "infer" or mtype == "predict_action":
+            try:
+                ouput_dict = self._policy.predict_action(**payload)
+            except Exception as e:
+                logging.exception("Policy inference error (request_id=%s)", req_id)
+                logging.exception(e)
+                
+                return {
+                    "status": "error",
+                    "ok": False,
+                    "type": "inference_result",
+                    "request_id": req_id,
+                    "error": {
+                        "message": str(e),
+                        # "traceback": traceback.format_exc(),
+                    },
+                }
+            data = ouput_dict
+            return {
+                "status": "ok",
+                "ok": True,
+                "type": "inference_result",
+                "request_id": req_id,
+                "data": data,
+            }
+
+        # unknow request type
+        else:
+            return {
+                "status": "error",
+                "ok": False,
+                "type": "unknown",
+                "request_id": req_id,
+                "error": {"message": f"Unsupported message type '{mtype}'"},
+            }
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO, force=True)
+    # Example usage:
+    # policy = YourPolicyClass()  # Replace with your actual policy class
+    # server = WebsocketPolicyServer(policy, host="localhost", port=10091)
+    # server.serve_forever()
+    raise NotImplementedError("This module is not intended to be run directly.")
+#
+#  Instead, it should be imported and used in a server context.

real_robot/source/deployment/model_server/xtrainer2_command.py

@@ -0,0 +1,93 @@
+"""xtrainer2 command layout used between StarVLA and the real robot.
+
+The converted LeRobot dataset stores each arm as ``xyz + quaternion + gripper``.
+StarVLA / PyTorch3D read that quaternion as ``w, x, y, z``.  The robot HDF5
+streams and the robot controller use ``x, y, z, w``.
+
+The training transform already maps quaternion <-> rotation-6D.  After
+un-normalization the server therefore has 16-D **wxyz** commands.  Sending
+those 16 numbers to the robot without this reorder will execute the wrong
+orientation.
+
+Do not confuse these widths:
+
+* 14  – joint angles in the source HDF5 (not used for this VLA)
+* 16  – stored / robot EEF command (after this conversion, xyzw)
+* 20  – model tensor after rotation-6D (horizon is a separate axis)
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+from starVLA.dataloader.relative_pose import compose_quaternion
+
+QUATERNION_STARTS = (3, 11)
+POSITION_SLICES = ((0, 3), (8, 11))
+ACTION_DIM = 16
+ROBOT_TYPE = "xtrainer2_bimanual_eef"
+SOURCE_QUATERNION_FORMAT = "xyzw"
+MODEL_QUATERNION_FORMAT = "wxyz"
+
+#: Value of ``DataConfig.robot_command_layout`` that selects the conversions in
+#: this module. The policy server reads that attribute instead of matching on a
+#: class name.
+COMMAND_LAYOUT = "xtrainer2_bimanual_eef_xyzw"
+
+
+def _as_float_array(values: np.ndarray) -> np.ndarray:
+    array = np.asarray(values, dtype=np.float32)
+    if array.shape[-1] != ACTION_DIM:
+        raise ValueError(f"expected last dimension {ACTION_DIM}, got {array.shape}")
+    return array
+
+
+def wxyz_to_xyzw_block(block: np.ndarray) -> np.ndarray:
+    """``w, x, y, z`` -> ``x, y, z, w``."""
+    return np.concatenate([block[..., 1:4], block[..., :1]], axis=-1)
+
+
+def xyzw_to_wxyz_block(block: np.ndarray) -> np.ndarray:
+    """``x, y, z, w`` -> ``w, x, y, z``."""
+    return np.concatenate([block[..., 3:4], block[..., :3]], axis=-1)
+
+
+def model_actions_to_robot(values: np.ndarray) -> np.ndarray:
+    """Convert un-normalized StarVLA actions (wxyz) to robot commands (xyzw)."""
+    converted = np.array(_as_float_array(values), copy=True)
+    for start in QUATERNION_STARTS:
+        converted[..., start : start + 4] = wxyz_to_xyzw_block(converted[..., start : start + 4])
+    return converted
+
+
+def robot_state_to_model(values: np.ndarray) -> np.ndarray:
+    """Convert robot observations (xyzw) to the layout StarVLA was trained on."""
+    converted = np.array(_as_float_array(values), copy=True)
+    for start in QUATERNION_STARTS:
+        converted[..., start : start + 4] = xyzw_to_wxyz_block(converted[..., start : start + 4])
+    return converted
+
+
+def compose_relative_eef_wxyz(relative_actions: np.ndarray, state_wxyz: np.ndarray) -> np.ndarray:
+    """Turn rel-mode 16-D wxyz actions into absolute 16-D wxyz poses.
+
+    Positions add the current EE; rotations left-multiply the current
+    quaternion; grippers are already absolute and are left unchanged.
+    """
+    relative = np.array(_as_float_array(relative_actions), copy=True)
+    state = np.asarray(state_wxyz, dtype=np.float32).reshape(-1, ACTION_DIM)[0]
+    absolute = relative
+    for start, end in POSITION_SLICES:
+        absolute[..., start:end] = relative[..., start:end] + state[start:end]
+    for start in QUATERNION_STARTS:
+        absolute[..., start : start + 4] = compose_quaternion(
+            state[start : start + 4],
+            relative[..., start : start + 4],
+        )
+    return absolute
+
+
+def is_relative_action_mode(action_mode: object | None) -> bool:
+    if action_mode is None:
+        return False
+    return str(action_mode).lower() in {"rel", "relative"}

real_robot/source/examples/Xtrainer2/INFERENCE.md

@@ -0,0 +1,4 @@
+# Xtrainer2 policy inference
+
+The server and client interface is described in [ROBOT.md](ROBOT.md).
+Data layout and training configuration are described in [README.md](README.md).

real_robot/source/examples/Xtrainer2/README.md

@@ -0,0 +1,26 @@
+# Xtrainer2 data adapter
+
+This directory contains optional StarVLA configurations and conversion helpers
+for Xtrainer2 LeRobot data. It does not distribute robot data, checkpoints, or
+on-robot control software. The ReTime command in the repository root is the
+recommended entrypoint for proposal generation and training.
+
+The supplied configurations cover pack-phone with absolute end-effector actions
+and pick-bottle with relative end-effector actions. For bimanual data, the source
+action contains 16 values: position, quaternion, and gripper for each arm.
+Rotation conversion expands the model action to 20 values. `pytorch3d` is
+required for this conversion; install a build matching your Python, PyTorch,
+and CUDA environment. The additional Python dependencies are listed in
+`requirements-xtrainer2.txt`.
+
+Set dataset, model, statistics, training budget, and process count for your own
+environment. Keep the trainer gradient accumulation equal to the Accelerate
+configuration. Use a full checkpoint directory with `--start resume` when
+optimizer and scheduler continuation is required. A model file with the default
+`--start weights` initializes a new optimizer and scheduler.
+
+Policy serving uses the included `deployment/model_server` implementation. A
+separate client must supply observations and consume actions using the same
+schema as the selected data configuration. Client source and deployment
+instructions are outside this artifact. See [ROBOT.md](ROBOT.md) for the data
+and interface contract.

real_robot/source/examples/Xtrainer2/ROBOT.md

@@ -0,0 +1,18 @@
+# Xtrainer2 policy interface
+
+The optional policy server is in `deployment/model_server`. Configure it with a
+model checkpoint and the matching training configuration and dataset statistics.
+The client is a separate component and is not distributed in this artifact.
+
+The client sends camera observations, proprioception, and a task instruction.
+For the supplied bimanual configurations, its proprioception and actions use
+16 absolute end-effector values: position, quaternion, and gripper for each arm.
+The model internally uses 20 values after quaternion to rotation-6D conversion.
+The client quaternion format is `xyzw`; dataset conversion and normalization
+must follow the selected configuration. The pick-bottle configuration composes
+relative actions on the server before returning absolute client actions.
+
+Check the model architecture, action dimension, normalization statistics,
+camera names, and action chunk size against the checkpoint before deployment.
+Only use a client whose message schema matches the server's documented policy
+interface. Robot hardware and control software are outside this review package.

real_robot/source/examples/Xtrainer2/train_files/build_pack_phone_support.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""Write frozen fingerprint and provenance for xtrainer2_pack_phone."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+
+def _identity_files(dataset_root: Path) -> list[Path]:
+    files = list((dataset_root / "data").rglob("*.parquet"))
+    files.extend((dataset_root / "videos").rglob("*.mp4"))
+    meta_root = dataset_root / "meta"
+    if meta_root.exists():
+        for pattern in ("*.json", "*.jsonl", "*.parquet"):
+            files.extend(meta_root.rglob(pattern))
+    return sorted(set(files), key=lambda path: path.relative_to(dataset_root).as_posix())
+
+
+def compute_dataset_content_fingerprint(dataset_root: str | Path) -> dict[str, Any]:
+    root = Path(dataset_root)
+    entries = []
+    total_bytes = 0
+    for file_path in _identity_files(root):
+        relative_path = file_path.relative_to(root).as_posix()
+        file_size = file_path.stat().st_size
+        entries.append({"path": relative_path, "bytes": file_size})
+        total_bytes += file_size
+    return {
+        "file_count": len(entries),
+        "total_bytes": total_bytes,
+        "files": entries,
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data-root",
+        type=Path,
+        default=Path("/path/to/data/lerobot"),
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=Path,
+        default=None,
+    )
+    args = parser.parse_args()
+    dataset_root = args.data_root / "xtrainer2_pack_phone"
+    output_dir = args.output_dir or (args.data_root / "xtrainer2_optr_support" / "pack_phone")
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    info = json.loads((dataset_root / "meta" / "info.json").read_text(encoding="utf-8"))
+    report_path = dataset_root / "conversion_report.json"
+    report = json.loads(report_path.read_text(encoding="utf-8")) if report_path.is_file() else {}
+
+    provenance = {
+        "dataset_name": "xtrainer2_pack_phone",
+        "codebase_version": info.get("codebase_version"),
+        "robot_type": info.get("robot_type"),
+        "fps": info.get("fps"),
+        "total_episodes": info.get("total_episodes"),
+        "total_frames": info.get("total_frames"),
+        "quaternion_format": info.get("quaternion_format"),
+        "source_quaternion_format": info.get("source_quaternion_format"),
+        "source_fingerprint": report.get("source_fingerprint"),
+    }
+    provenance_path = output_dir / "data_provenance.json"
+    provenance_path.write_text(json.dumps(provenance, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+
+    fingerprint = {
+        "datasets": {
+            "xtrainer2_pack_phone": compute_dataset_content_fingerprint(dataset_root),
+        }
+    }
+    fingerprint_path = output_dir / "content_fingerprints.json"
+    fingerprint_path.write_text(json.dumps(fingerprint) + "\n", encoding="utf-8")
+
+    print(f"wrote {provenance_path}")
+    print(f"wrote {fingerprint_path}")
+
+
+if __name__ == "__main__":
+    main()

real_robot/source/examples/Xtrainer2/train_files/build_pick_bottle_support.py

@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Write frozen fingerprint and provenance for xtrainer2_pick_bottle."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+
+def _identity_files(dataset_root: Path) -> list[Path]:
+    files = list((dataset_root / "data").rglob("*.parquet"))
+    files.extend((dataset_root / "videos").rglob("*.mp4"))
+    meta_root = dataset_root / "meta"
+    if meta_root.exists():
+        for pattern in ("*.json", "*.jsonl", "*.parquet"):
+            files.extend(meta_root.rglob(pattern))
+    return sorted(set(files), key=lambda path: path.relative_to(dataset_root).as_posix())
+
+
+def compute_dataset_content_fingerprint(dataset_root: str | Path) -> dict[str, Any]:
+    root = Path(dataset_root)
+    entries = []
+    total_bytes = 0
+    for file_path in _identity_files(root):
+        relative_path = file_path.relative_to(root).as_posix()
+        file_size = file_path.stat().st_size
+        entries.append({"path": relative_path, "bytes": file_size})
+        total_bytes += file_size
+    return {
+        "file_count": len(entries),
+        "total_bytes": total_bytes,
+        "files": entries,
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data-root",
+        type=Path,
+        default=Path("/path/to/data/lerobot"),
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=Path,
+        default=None,
+    )
+    args = parser.parse_args()
+    dataset_root = args.data_root / "xtrainer2_pick_bottle"
+    output_dir = args.output_dir or (args.data_root / "xtrainer2_optr_support" / "pick_bottle")
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    info = json.loads((dataset_root / "meta" / "info.json").read_text(encoding="utf-8"))
+    report_path = dataset_root / "conversion_report.json"
+    report = json.loads(report_path.read_text(encoding="utf-8")) if report_path.is_file() else {}
+
+    provenance = {
+        "dataset_name": "xtrainer2_pick_bottle",
+        "codebase_version": info.get("codebase_version"),
+        "robot_type": info.get("robot_type"),
+        "fps": info.get("fps"),
+        "total_episodes": info.get("total_episodes"),
+        "total_frames": info.get("total_frames"),
+        "total_tasks": info.get("total_tasks"),
+        "quaternion_format": info.get("quaternion_format"),
+        "source_quaternion_format": info.get("source_quaternion_format"),
+        "source_fingerprint": report.get("source_fingerprint"),
+    }
+    provenance_path = output_dir / "data_provenance.json"
+    provenance_path.write_text(json.dumps(provenance, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+
+    fingerprint = {
+        "datasets": {
+            "xtrainer2_pick_bottle": compute_dataset_content_fingerprint(dataset_root),
+        }
+    }
+    fingerprint_path = output_dir / "content_fingerprints.json"
+    fingerprint_path.write_text(json.dumps(fingerprint) + "\n", encoding="utf-8")
+
+    print(f"wrote {provenance_path}")
+    print(f"wrote {fingerprint_path}")
+
+
+if __name__ == "__main__":
+    main()

real_robot/source/examples/Xtrainer2/train_files/data_registry/data_config.py

@@ -0,0 +1,179 @@
+"""StarVLA data registry for xtrainer2 bimanual EEF datasets."""
+
+from starVLA.dataloader.gr00t_lerobot.datasets import ModalityConfig
+from starVLA.dataloader.gr00t_lerobot.embodiment_tags import EmbodimentTag
+from starVLA.dataloader.gr00t_lerobot.transform.base import ComposedModalityTransform
+from starVLA.dataloader.gr00t_lerobot.transform.state_action import (
+    StateActionToTensor,
+    StateActionTransform,
+)
+from starVLA.dataloader.gr00t_lerobot.transform.video import (
+    VideoColorJitter,
+    VideoCrop,
+    VideoResize,
+    VideoToNumpy,
+    VideoToTensor,
+)
+
+
+class Xtrainer2BimanualEefDataConfig:
+    """Three-camera bimanual EEF control at 10 Hz.
+
+    LeRobot stores 16 raw dimensions per state/action in wxyz.  Training
+    converts quaternions to rotation-6D (20 model dims) and inverts them
+    back to 16-D wxyz.  With ``action_mode: rel`` the dataloader expresses
+    xyz and orientation relative to the current EE; the websocket server
+    composes those back to absolute poses and reorders wxyz -> xyzw.
+    See ``examples/Xtrainer2/README.md``.
+    """
+
+    embodiment_tag = EmbodimentTag.NEW_EMBODIMENT
+    video_keys = [
+        "video.cam_high",
+        "video.cam_left_wrist",
+        "video.cam_right_wrist",
+    ]
+    state_keys = [
+        "state.left_position",
+        "state.left_rotation",
+        "state.left_gripper",
+        "state.right_position",
+        "state.right_rotation",
+        "state.right_gripper",
+    ]
+    action_keys = [
+        "action.left_position",
+        "action.left_rotation",
+        "action.left_gripper",
+        "action.right_position",
+        "action.right_rotation",
+        "action.right_gripper",
+    ]
+    language_keys = ["annotation.human.action.task_description"]
+    observation_indices = [0]
+    action_indices = list(range(16))
+
+    # Raw widths, matching ``meta/modality.json`` and the 16-D arrays in
+    # ``dataset_statistics.json``.  16 is not divisible by the 6 keys, so the
+    # policy server cannot infer the split and must be told.
+    state_key_dims = {
+        "state.left_position": 3,
+        "state.left_rotation": 4,
+        "state.left_gripper": 1,
+        "state.right_position": 3,
+        "state.right_rotation": 4,
+        "state.right_gripper": 1,
+    }
+    action_key_dims = {
+        "action.left_position": 3,
+        "action.left_rotation": 4,
+        "action.left_gripper": 1,
+        "action.right_position": 3,
+        "action.right_rotation": 4,
+        "action.right_gripper": 1,
+    }
+    # Source representation of every rotation key.  The dataset carries this in
+    # ``meta/modality.json``; deployment rebuilds the transforms from the
+    # checkpoint alone, where that file is not available.
+    rotation_types = {
+        "state.left_rotation": "quaternion",
+        "state.right_rotation": "quaternion",
+        "action.left_rotation": "quaternion",
+        "action.right_rotation": "quaternion",
+    }
+
+    # Deployment contract (read by deployment/model_server).  The robot sends
+    # and receives 16 raw dims with xyzw quaternions; the model works in 20
+    # normalized dims with wxyz.  See deployment/model_server/xtrainer2_command.py.
+    robot_command_layout = "xtrainer2_bimanual_eef_xyzw"
+    server_normalizes_state = True
+
+    def modality_config(self):
+        return {
+            "video": ModalityConfig(
+                delta_indices=self.observation_indices,
+                modality_keys=self.video_keys,
+            ),
+            "state": ModalityConfig(
+                delta_indices=self.observation_indices,
+                modality_keys=self.state_keys,
+            ),
+            "action": ModalityConfig(
+                delta_indices=self.action_indices,
+                modality_keys=self.action_keys,
+            ),
+            "language": ModalityConfig(
+                delta_indices=self.observation_indices,
+                modality_keys=self.language_keys,
+            ),
+        }
+
+    def transform(self):
+        rotation_keys = {
+            "state.left_rotation": "rotation_6d",
+            "state.right_rotation": "rotation_6d",
+        }
+        action_rotation_keys = {
+            "action.left_rotation": "rotation_6d",
+            "action.right_rotation": "rotation_6d",
+        }
+        state_normalization = {
+            key: "min_max" if key.endswith("_rotation") else "q99"
+            for key in self.state_keys
+        }
+        action_normalization = {
+            key: "min_max" if key.endswith("_rotation") else "q99"
+            for key in self.action_keys
+        }
+        return ComposedModalityTransform(
+            transforms=[
+                VideoToTensor(apply_to=self.video_keys),
+                VideoCrop(apply_to=self.video_keys, scale=0.95),
+                VideoResize(
+                    apply_to=self.video_keys,
+                    height=224,
+                    width=224,
+                    interpolation="linear",
+                ),
+                VideoColorJitter(
+                    apply_to=self.video_keys,
+                    brightness=0.3,
+                    contrast=0.4,
+                    saturation=0.5,
+                    hue=0.08,
+                ),
+                VideoToNumpy(apply_to=self.video_keys),
+                StateActionToTensor(apply_to=self.state_keys),
+                StateActionTransform(
+                    apply_to=self.state_keys,
+                    normalization_modes=state_normalization,
+                    target_rotations=rotation_keys,
+                ),
+                StateActionToTensor(apply_to=self.action_keys),
+                StateActionTransform(
+                    apply_to=self.action_keys,
+                    normalization_modes=action_normalization,
+                    target_rotations=action_rotation_keys,
+                ),
+            ]
+        )
+
+
+ROBOT_TYPE_CONFIG_MAP = {
+    "xtrainer2_bimanual_eef": Xtrainer2BimanualEefDataConfig(),
+}
+
+ROBOT_TYPE_TO_EMBODIMENT_TAG = {}
+
+DATASET_NAMED_MIXTURES = {
+    "xtrainer2_pick_bottle": [
+        ("xtrainer2_pick_bottle", 1.0, "xtrainer2_bimanual_eef"),
+    ],
+    "xtrainer2_pack_phone": [
+        ("xtrainer2_pack_phone", 1.0, "xtrainer2_bimanual_eef"),
+    ],
+    "xtrainer2_pick_pack": [
+        ("xtrainer2_pick_bottle", 1.0, "xtrainer2_bimanual_eef"),
+        ("xtrainer2_pack_phone", 1.0, "xtrainer2_bimanual_eef"),
+    ],
+}

real_robot/source/examples/Xtrainer2/train_files/modality.json

@@ -0,0 +1,26 @@
+{
+  "state": {
+    "left_position": {"start": 0, "end": 3, "dtype": "float32", "absolute": true, "original_key": "observation.state"},
+    "left_rotation": {"start": 3, "end": 7, "dtype": "float32", "absolute": true, "rotation_type": "quaternion", "original_key": "observation.state"},
+    "left_gripper": {"start": 7, "end": 8, "dtype": "float32", "absolute": true, "original_key": "observation.state"},
+    "right_position": {"start": 8, "end": 11, "dtype": "float32", "absolute": true, "original_key": "observation.state"},
+    "right_rotation": {"start": 11, "end": 15, "dtype": "float32", "absolute": true, "rotation_type": "quaternion", "original_key": "observation.state"},
+    "right_gripper": {"start": 15, "end": 16, "dtype": "float32", "absolute": true, "original_key": "observation.state"}
+  },
+  "action": {
+    "left_position": {"start": 0, "end": 3, "dtype": "float32", "absolute": true, "original_key": "action"},
+    "left_rotation": {"start": 3, "end": 7, "dtype": "float32", "absolute": true, "rotation_type": "quaternion", "original_key": "action"},
+    "left_gripper": {"start": 7, "end": 8, "dtype": "float32", "absolute": true, "original_key": "action"},
+    "right_position": {"start": 8, "end": 11, "dtype": "float32", "absolute": true, "original_key": "action"},
+    "right_rotation": {"start": 11, "end": 15, "dtype": "float32", "absolute": true, "rotation_type": "quaternion", "original_key": "action"},
+    "right_gripper": {"start": 15, "end": 16, "dtype": "float32", "absolute": true, "original_key": "action"}
+  },
+  "video": {
+    "cam_high": {"original_key": "observation.images.cam_high"},
+    "cam_left_wrist": {"original_key": "observation.images.cam_left_wrist"},
+    "cam_right_wrist": {"original_key": "observation.images.cam_right_wrist"}
+  },
+  "annotation": {
+    "human.action.task_description": {"original_key": "task_index"}
+  }
+}

real_robot/source/examples/Xtrainer2/train_files/run_xtrainer2_policy_server.sh

@@ -0,0 +1,111 @@
+#!/usr/bin/env bash
+# Start the StarVLA websocket policy server for xtrainer2.
+# The robot PC talks to this process; do not clone RoboticsX into this repo.
+# The on-robot client is a separate component.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
+export PYTHONPATH="${REPO_ROOT}:${PYTHONPATH:-}"
+
+if [[ -f "${REPO_ROOT}/.env.xtrainer2-server" ]]; then
+  # Local GPU-server paths (gitignored). Sets OPTR_PYTHON / FLORENCE_MODEL_PATH.
+  # shellcheck disable=SC1091
+  source "${REPO_ROOT}/.env.xtrainer2-server"
+fi
+
+if [[ $# -lt 1 ]]; then
+  echo "Usage: bash examples/Xtrainer2/train_files/run_xtrainer2_policy_server.sh <ckpt.pt> [gpu_id] [port]" >&2
+  exit 1
+fi
+
+ckpt=$(readlink -f "$1")
+gpu_id=${2:-0}
+port=${3:-10093}
+if [[ -z "${OPTR_PYTHON:-}" && -z "${STARVLA_PYTHON:-}" && -x "${REPO_ROOT}/.venv-xtrainer2/bin/python" ]]; then
+  python_bin="${REPO_ROOT}/.venv-xtrainer2/bin/python"
+else
+  python_bin=${OPTR_PYTHON:-${STARVLA_PYTHON:-python}}
+fi
+
+if [[ ! -f $ckpt ]]; then
+  echo "checkpoint not found: ${ckpt}" >&2
+  exit 1
+fi
+
+# `read_mode_config` resolves config.yaml and dataset_statistics.json from the
+# checkpoint's grandparent, i.e. it expects <run>/checkpoints/<name>.pt. A
+# DeepSpeed run instead writes <run>/checkpoints/steps_N_accelerate/pytorch_model.pt,
+# which is one level deeper, so stage that layout out of the way of the run dir.
+ckpt_dir=$(dirname "$ckpt")
+if [[ -f "${ckpt_dir}/../config.yaml" ]]; then
+  run_dir=$(cd "${ckpt_dir}/.." && pwd)
+  serve_ckpt=$ckpt
+  stage_dir=""
+elif [[ -f "${ckpt_dir}/../../config.yaml" ]]; then
+  run_dir=$(cd "${ckpt_dir}/../.." && pwd)
+  stage_dir="${TMPDIR:-/tmp}/starvla-serve/$(basename "$run_dir")/$(basename "$ckpt_dir")"
+  mkdir -p "${stage_dir}/checkpoints"
+  ln -sfn "${run_dir}/config.yaml" "${stage_dir}/config.yaml"
+  ln -sfn "${run_dir}/dataset_statistics.json" "${stage_dir}/dataset_statistics.json"
+  ln -sfn "$ckpt" "${stage_dir}/checkpoints/$(basename "$ckpt_dir")_pytorch_model.pt"
+  serve_ckpt="${stage_dir}/checkpoints/$(basename "$ckpt_dir")_pytorch_model.pt"
+  echo "[INFO] accelerate checkpoint layout; serving through ${stage_dir}"
+else
+  echo "need config.yaml and dataset_statistics.json in the run dir above ${ckpt_dir}" >&2
+  exit 1
+fi
+
+# Training config.yaml records the cluster Florence path. Rewrite it locally
+# so from_pretrained does not look for /path/to/models/... .
+if [[ -n "${FLORENCE_MODEL_PATH:-}" ]]; then
+  if [[ ! -d $FLORENCE_MODEL_PATH ]]; then
+    echo "FLORENCE_MODEL_PATH is not a directory: ${FLORENCE_MODEL_PATH}" >&2
+    exit 1
+  fi
+  if [[ -z $stage_dir ]]; then
+    stage_dir="${TMPDIR:-/tmp}/starvla-serve/$(basename "$run_dir")/local-florence"
+    mkdir -p "${stage_dir}/checkpoints"
+    ln -sfn "$ckpt" "${stage_dir}/checkpoints/$(basename "$ckpt")"
+    ln -sfn "${run_dir}/dataset_statistics.json" "${stage_dir}/dataset_statistics.json"
+    serve_ckpt="${stage_dir}/checkpoints/$(basename "$ckpt")"
+  fi
+  FLORENCE_MODEL_PATH="$FLORENCE_MODEL_PATH" "$python_bin" - "${run_dir}/config.yaml" "${stage_dir}/config.yaml" <<'PY'
+import os
+import sys
+
+from omegaconf import OmegaConf
+
+src, dst = sys.argv[1], sys.argv[2]
+path = os.environ["FLORENCE_MODEL_PATH"]
+if "florence" not in path.lower():
+    raise SystemExit(f"FLORENCE_MODEL_PATH must contain 'florence': {path}")
+cfg = OmegaConf.load(src)
+cfg.framework.qwenvl.base_vlm = path
+OmegaConf.save(cfg, dst)
+print(f"[INFO] rewrote base_vlm -> {path}")
+PY
+fi
+
+if [[ ! -f $run_dir/dataset_statistics.json ]]; then
+  echo "missing ${run_dir}/dataset_statistics.json" >&2
+  exit 1
+fi
+
+echo "[INFO] xtrainer2 policy server"
+echo "[INFO] checkpoint: ${ckpt}"
+echo "[INFO] run dir:    ${run_dir}"
+echo "[INFO] gpu: ${gpu_id}  port: ${port}"
+echo "[INFO] client contract: 3 images (cam_high, cam_left_wrist, cam_right_wrist)"
+echo "[INFO]   HWC RGB uint8, 16-D robot state and 16-D actions, quaternions xyzw"
+echo "[INFO] instruction must be English: Pick up the phone, place it in the box, and close the lid."
+
+# Local Florence weights; do not send Hub traffic through a lab HTTP proxy.
+unset HF_ENDPOINT || true
+idle_timeout="${OPTR_IDLE_TIMEOUT:--1}"
+CUDA_VISIBLE_DEVICES=$gpu_id "$python_bin" \
+  "$REPO_ROOT/deployment/model_server/server_policy.py" \
+  --ckpt_path "$serve_ckpt" \
+  --port "$port" \
+  --use_bf16 \
+  --idle_timeout "$idle_timeout"

real_robot/source/examples/Xtrainer2/train_files/run_xtrainer2_train.sh

@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if [[ $# -ne 2 ]]; then
+  echo "usage: $0 <config.yaml> <run-id>" >&2
+  exit 64
+fi
+
+project="${OPTR_PROJECT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)}"
+python="${OPTR_PYTHON:-python}"
+accelerate_config="${OPTR_ACCELERATE_CONFIG:-${project}/configs/deepspeed_florence_8gpu.yaml}"
+num_processes="${OPTR_NUM_PROCESSES:-8}"
+config=$1
+run_id=$2
+
+if [[ ! -f "$config" && -f "${project}/${config}" ]]; then
+  config="${project}/${config}"
+fi
+if [[ ! -f "$config" ]]; then
+  echo "error: training config not found: ${config}" >&2
+  exit 66
+fi
+
+# The ${oc.env:...} refs in the training yaml have no defaults; OmegaConf
+# resolves them eagerly and would abort on all ranks. Collect names from the
+# file so pack_phone and pick_bottle (different provenance vars) share this
+# launcher.
+mapfile -t required_vars < <("$python" - "$config" <<'PY'
+import re
+import sys
+from pathlib import Path
+
+text = Path(sys.argv[1]).read_text(encoding="utf-8")
+names = sorted(set(re.findall(r"\$\{oc\.env:([A-Za-z_][A-Za-z0-9_]*)\}", text)))
+if not names:
+    sys.exit("error: no ${oc.env:...} interpolations found in " + sys.argv[1])
+print("\n".join(names))
+PY
+)
+
+missing=()
+for var in "${required_vars[@]}"; do
+  if [[ -z "${!var:-}" ]]; then
+    missing+=("$var")
+  fi
+done
+if (( ${#missing[@]} )); then
+  echo "error: required environment variables are unset: ${missing[*]}" >&2
+  echo "see examples/Xtrainer2/README.md for the export block" >&2
+  exit 78
+fi
+
+if [[ -n "${FLORENCE_MODEL_PATH:-}" ]]; then
+  if [[ "${FLORENCE_MODEL_PATH,,}" != *florence* ]]; then
+    echo "error: FLORENCE_MODEL_PATH must contain 'florence' or the VLM dispatch" >&2
+    echo "       raises NotImplementedError: ${FLORENCE_MODEL_PATH}" >&2
+    exit 78
+  fi
+  if [[ ! -d "$FLORENCE_MODEL_PATH" ]]; then
+    echo "error: FLORENCE_MODEL_PATH is not a directory: ${FLORENCE_MODEL_PATH}" >&2
+    exit 66
+  fi
+fi
+
+for var in "${required_vars[@]}"; do
+  value="${!var}"
+  case "$var" in
+    FLORENCE_MODEL_PATH|OPTR_RUN_ROOT|XTRAINER2_DATA_ROOT) continue ;;
+  esac
+  if [[ "$var" == *_PATH || "$var" == *_DIR ]]; then
+    if [[ ! -e "$value" ]]; then
+      echo "error: support file missing: ${value} (${var})" >&2
+      echo "       run the matching examples/Xtrainer2/train_files/build_*_support.py first" >&2
+      exit 66
+    fi
+  fi
+done
+
+if ! command -v "$python" >/dev/null 2>&1 && [[ ! -x "$python" ]]; then
+  echo "error: python interpreter not found: ${python}" >&2
+  echo "       set OPTR_PYTHON to an interpreter with deepspeed and pytorch3d" >&2
+  exit 69
+fi
+
+if [[ ! -f "$accelerate_config" ]]; then
+  echo "error: accelerate config not found: ${accelerate_config}" >&2
+  exit 66
+fi
+
+cd "$project"
+export PYTHONPATH="${project}:${PYTHONPATH:-}"
+export WANDB_MODE="${WANDB_MODE:-offline}"
+"$python" -m accelerate.commands.launch \
+  --config_file "$accelerate_config" \
+  --num_processes "$num_processes" \
+  starVLA/training/train_starvla.py \
+  --config_yaml "$config" \
+  --run_id "$run_id"

real_robot/source/licenses/Diffusion-Policy-LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Columbia Artificial Intelligence and Robotics Lab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

real_robot/source/licenses/Isaac-GR00T-LICENSE

@@ -0,0 +1,190 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to the Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by the Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding any notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

real_robot/source/licenses/MODIFIED_FILES.txt

@@ -0,0 +1,17 @@
+starVLA/dataloader/lerobot_datasets.py
+starVLA/dataloader/__init__.py
+starVLA/training/train_starvla.py
+starVLA/training/trainer_utils/overwatch.py
+starVLA/training/trainer_utils/trainer_tools.py
+starVLA/model/framework/base_framework.py
+starVLA/model/framework/share_tools.py
+starVLA/model/modules/action_model/GR00T_ActionHeader.py
+starVLA/model/modules/action_model/__init__.py
+starVLA/model/modules/vlm/Florence2.py
+starVLA/model/modules/vlm/QWen3.py
+starVLA/model/modules/vlm/__init__.py
+starVLA/model/modules/action_model/flow_matching_head/cross_attention_dit.py
+starVLA/model/framework/VLM4A/QwenGR00T.py
+starVLA/dataloader/gr00t_lerobot/datasets.py
+starVLA/dataloader/gr00t_lerobot/data_config.py
+starVLA/dataloader/gr00t_lerobot/transform/state_action.py

real_robot/source/licenses/OpenVLA-LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

real_robot/source/licenses/X-VLA-LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

real_robot/source/licenses/msgpack-numpy-LICENSE

@@ -0,0 +1,35 @@
+<!---
+-*- mode:markdown -*-
+vi:ft=markdown
+-->
+License
+=======
+
+Copyright (c) 2013-2022, Lev E. Givon.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+* Redistributions of source code must retain the above copyright
+  notice, this list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above
+  copyright notice, this list of conditions and the following
+  disclaimer in the documentation and/or other materials provided
+  with the distribution.
+* Neither the name of Lev E. Givon nor the names of any
+  contributors may be used to endorse or promote products derived
+  from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

real_robot/source/pyproject.toml

@@ -0,0 +1,81 @@
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "retime-starvla"
+description = "ReTime temporal sampling plugin and compatible StarVLA backend."
+version = "0.1.0"
+readme = "README.md"
+requires-python = ">=3.10"
+keywords = ["vision-language-action models", "robot learning", "importance sampling"]
+license = {file = "LICENSE"}
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3 :: Only",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["omegaconf>=2.3,<3", "numpy", "scipy", "torch"]
+
+[project.scripts]
+retime = "retime.cli:main"
+retime-pretrain = "retime.pretrain:main"
+
+[project.optional-dependencies]
+dev = [
+    "black>=24.2.0",
+    "ruff>=0.2.2",
+]
+
+
+[tool.setuptools]
+license-files = ["LICENSE", "NOTICE", "licenses/*", "third_party/frameskip/LICENSE"]
+
+[tool.setuptools.packages.find]
+where = ["."]
+exclude = [
+    "tests",
+    "tests.*",
+    "cache",
+    "cache.*",
+    "playground",
+    "playground.*",
+    "results",
+    "results.*",
+    "checkpoints",
+    "checkpoints.*",
+    "scripts",
+    "scripts.*",
+    "eval",
+    "eval.*",
+    "assets",
+    "assets.*",
+    ".vscode",
+    "*.egg-info",
+    "**__pycache__**",
+    "*.pyc",
+    "*.pyo",
+    "starVLA.egg-info",
+    "starVLA.egg-info.*",
+    "starVLA.config",
+    "starVLA.config.*",
+]
+
+[tool.black]
+line-length = 121
+target-version = ["py310"]
+preview = true
+
+[tool.ruff]
+line-length = 121
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["A", "B", "E", "F", "I", "RUF", "W"]
+ignore = ["F722"]
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = ["E402", "F401"]

real_robot/source/requirements-qwen.txt

@@ -0,0 +1,9 @@
+# Optional dependencies for the supplied Qwen-GR00T configurations.
+# Install requirements.txt first. Qwen Bridge and Qwen RoboTwin are included in
+# the GPU integration test documented in VALIDATION.md.
+
+-r requirements.txt
+bitsandbytes==0.50.0
+packaging
+psutil
+ninja

real_robot/source/requirements-xtrainer2.txt

@@ -0,0 +1,10 @@
+# Optional dependencies for Xtrainer2 data processing and policy serving.
+# Install requirements.txt first. This extension was not included in the
+# Florence Bridge GPU integration test documented in VALIDATION.md.
+#
+# Xtrainer2 action_dim=20 requires a PyTorch3D build compatible with the
+# installed Python, PyTorch, and CUDA versions. Install it separately from
+# its official distribution for your platform.
+
+-r requirements.txt
+pytest

real_robot/source/requirements.txt

@@ -0,0 +1,34 @@
+transformers==4.57.0
+accelerate==1.13.0
+tiktoken
+einops
+transformers_stream_generator==0.0.4
+scipy
+torch==2.7.1
+torchvision==0.22.1
+setuptools==80.9.0
+pillow
+tensorboard
+matplotlib
+albumentations==1.4.18
+decord==0.6.0
+pydantic==2.10.6
+pyarrow==14.0.1
+fastparquet==2024.11.0
+av==12.3.0
+numpydantic==1.6.9
+deepspeed==0.19.2
+qwen-vl-utils
+omegaconf
+numpy==1.26.4
+wandb
+rich
+diffusers==0.33.1
+timm
+tyro
+websockets==15.0.1
+msgpack==1.1.0
+tdigest==0.5.2.2
+
+# Qwen configurations also require requirements-qwen.txt and a compatible
+# Flash Attention 2 installation; see README.md.

real_robot/source/retime/__init__.py

@@ -0,0 +1,6 @@
+"""ReTime: objective-preserving temporal sampling for StarVLA.
+
+Importing the plugin or its CLI does not initialize a model or CUDA.
+"""
+
+__version__ = "0.1.0"

real_robot/source/retime/__main__.py

@@ -0,0 +1,4 @@
+from .cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

real_robot/source/retime/checkpoint.py

@@ -0,0 +1,19 @@
+"""Load full policy weights before or after DeepSpeed partitions parameters."""
+
+
+def load_policy_state_dict(model, state_dict, *, strict=True):
+    """Every distributed rank must call this with the same state dictionary.
+
+    ZeRO-3 initialization replaces parameter storage with empty shards. Gather
+    those parameters while copying weights, then broadcast and repartition on
+    context exit. Loading on every rank also initializes replicated buffers.
+    Ordinary models do not need DeepSpeed installed.
+    """
+    partitioned = [parameter for parameter in model.parameters() if hasattr(parameter, "ds_id")]
+    if not partitioned:
+        return model.load_state_dict(state_dict, strict=strict)
+
+    from deepspeed import zero
+
+    with zero.GatheredParameters(partitioned, modifier_rank=0):
+        return model.load_state_dict(state_dict, strict=strict)

real_robot/source/retime/cli.py

@@ -0,0 +1,272 @@
+"""One-command adapter to the accompanying StarVLA training backend.
+
+The CLI prepares temporal proposals and dispatches StarVLA training.
+The sampler and ordinary importance correction are unchanged.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+from pathlib import Path
+import subprocess
+import sys
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser(description=__doc__)
+    result.add_argument("--config", type=Path, required=True)
+    result.add_argument("--base-policy", type=Path, required=True,
+                        help="Complete Accelerate checkpoint directory, committed single-process model file, or model file with --start weights.")
+    result.add_argument("--start", choices=("resume", "weights"), default="weights",
+                        help="resume restores the full state; weights explicitly starts a new optimizer/schedule.")
+    result.add_argument("--alpha", type=float, default=0.2)
+    result.add_argument("--K", "--k", dest="k", type=int, default=8)
+    result.add_argument("--mode", nargs="+", choices=("retime", "uniform", "shuffle"),
+                        default=["retime"])
+    result.add_argument("--manifest", type=Path)
+    result.add_argument("--data-root", type=Path, help="Override the dataset root in the config")
+    result.add_argument("--data-mix", type=str, help="Override the named dataset mixture in the config")
+    result.add_argument("--vlm-path", type=Path, help="Override the Florence/Qwen backbone location")
+    result.add_argument("--statistics-path", type=Path, help="Override precomputed dataset statistics, if required")
+    result.add_argument(
+        "--omit-data-identity-files",
+        action="store_true",
+        help="Use the manifest's computed dataset identity without separate provenance/fingerprint files.",
+    )
+    result.add_argument("--epsilon", type=float, default=0.1)
+    result.add_argument("--shuffle-seed", type=int, default=12345)
+    result.add_argument("--accelerate-config", type=Path, required=True)
+    result.add_argument("--output", type=Path, required=True)
+    result.add_argument("--enable-wandb", action="store_true")
+    result.add_argument("--backend-root", type=Path, default=Path(__file__).resolve().parents[1],
+                        help="ReTime-enabled StarVLA source tree shipped with this artifact.")
+    result.add_argument("--dry-run", action="store_true",
+                        help="Validate local inputs and print commands; do not score, train, or write files.")
+    return result
+
+
+def _file(path: Path, label: str) -> Path:
+    path = path.expanduser().resolve()
+    if not path.is_file():
+        raise ValueError(f"{label} is not a file: {path}")
+    return path
+
+
+def _directory(path: Path, label: str) -> Path:
+    path = path.expanduser().resolve()
+    if not path.is_dir():
+        raise ValueError(f"{label} is not a directory: {path}")
+    return path
+
+
+def _native_commit_path(model_path: Path) -> Path:
+    for suffix in ("_pytorch_model.pt", "_model.safetensors"):
+        if model_path.name.endswith(suffix):
+            return model_path.with_name(
+                model_path.name[: -len(suffix)] + "_checkpoint.json"
+            )
+    raise ValueError(
+        "A single-process resume checkpoint must end in "
+        "_pytorch_model.pt or _model.safetensors"
+    )
+
+
+def _validate_resume_checkpoint(base: Path, world_size: int) -> tuple[Path, dict]:
+    if base.is_dir():
+        model = _file(base / "pytorch_model.pt", "base-policy model")
+        commit = json.loads(
+            _file(base / "checkpoint.json", "checkpoint commit").read_text(
+                encoding="utf-8"
+            )
+        )
+        if int(commit.get("world_size", -1)) != world_size:
+            raise ValueError("Base checkpoint world size differs from the launch contract")
+        entries = commit.get("files", [])
+        if not entries or any(
+            not (base / entry["name"]).is_file()
+            or (base / entry["name"]).stat().st_size != int(entry["bytes"])
+            for entry in entries
+        ):
+            raise ValueError("Base Accelerate checkpoint is incomplete")
+        return model, commit
+
+    model = _file(base, "base-policy model")
+    if world_size != 1:
+        raise ValueError(
+            "A committed single-process checkpoint requires a one-process launch"
+        )
+    commit_path = _native_commit_path(model)
+    commit = json.loads(_file(commit_path, "checkpoint commit").read_text(encoding="utf-8"))
+    files = commit.get("files", {})
+    for label in ("model", "training_state"):
+        entry = files.get(label, {})
+        path = model.parent / str(entry.get("name", ""))
+        if not path.is_file() or path.stat().st_size != int(entry.get("bytes", -1)):
+            raise ValueError(f"Base single-process checkpoint has an invalid {label} file")
+    if files["model"]["name"] != model.name:
+        raise ValueError("Checkpoint commit does not describe --base-policy")
+    return model, commit
+
+
+def build_plan(args: argparse.Namespace) -> dict:
+    from omegaconf import OmegaConf
+
+    if not math.isfinite(args.alpha) or not 0 <= args.alpha < 1:
+        raise ValueError("alpha must satisfy 0 <= alpha < 1")
+    if args.k < 1 or not math.isfinite(args.epsilon) or args.epsilon <= 0:
+        raise ValueError("K and epsilon must be positive")
+    if len(set(args.mode)) != len(args.mode):
+        raise ValueError("Each mode can be requested only once")
+    root = args.backend_root.expanduser().resolve()
+    config = _file(args.config, "config")
+    launch_config = _file(args.accelerate_config, "accelerate config")
+    cfg = OmegaConf.load(config)
+    if not args.enable_wandb:
+        cfg.wandb_project = None
+    launch = OmegaConf.load(launch_config)
+    output = args.output.expanduser().resolve()
+    cfg.run_root_dir = str(output / "runs")
+    cfg.output_dir = str(output / "prepare")
+    if args.data_root:
+        cfg.datasets.vla_data.data_root_dir = str(_directory(args.data_root, "dataset root"))
+    if args.data_mix:
+        cfg.datasets.vla_data.data_mix = args.data_mix
+    if args.vlm_path:
+        cfg.framework.qwenvl.base_vlm = str(_directory(args.vlm_path, "VLM backbone"))
+    if args.statistics_path:
+        cfg.datasets.vla_data.precomputed_statistics_path = str(_file(args.statistics_path, "statistics"))
+    if args.omit_data_identity_files:
+        cfg.datasets.vla_data.dataset_provenance_path = None
+        cfg.datasets.vla_data.dataset_content_fingerprint_path = None
+    # Model-only initialization derives support from the current dataset.
+    if args.start == "weights":
+        cfg.trainer.is_resume = False
+        cfg.trainer.resume_accelerate_checkpoint = None
+        cfg.trainer.resume_checkpoint_integrity_manifest = None
+        if OmegaConf.select(cfg, "trainer.initial_completed_steps") is not None:
+            cfg.trainer.initial_completed_steps = 0
+        if args.manifest is None:
+            cfg.datasets.vla_data.valid_anchor_manifest_path = None
+            cfg.datasets.vla_data.dataset_provenance_path = None
+            cfg.datasets.vla_data.dataset_content_fingerprint_path = None
+    cfg.datasets.vla_data.temporal_proposal_path = None
+    if args.manifest is not None:
+        cfg.datasets.vla_data.valid_anchor_manifest_path = str(args.manifest.expanduser().resolve())
+    if OmegaConf.select(cfg, "framework.name") != "QwenGR00T":
+        raise ValueError("This adapter supports the supplied Florence/Qwen GR00T configs only")
+    if not OmegaConf.select(cfg, "datasets.vla_data.traceable_sampling", default=False):
+        raise ValueError("ReTime requires traceable_sampling and a frozen valid-anchor manifest")
+    world_size = int(launch.num_processes)
+    if int(OmegaConf.select(launch, "num_machines", default=1)) != 1:
+        raise ValueError("The one-command launcher currently supports one machine")
+    if world_size != int(cfg.datasets.vla_data.expected_world_size):
+        raise ValueError("Accelerate process count differs from the dataset training contract")
+    accumulation = int(cfg.trainer.gradient_accumulation_steps)
+    ds_accumulation = OmegaConf.select(launch, "deepspeed_config.gradient_accumulation_steps")
+    if ds_accumulation is not None and int(ds_accumulation) != accumulation:
+        raise ValueError("Accelerate and trainer gradient accumulation differ")
+    batch = int(cfg.datasets.vla_data.per_device_batch_size)
+    if min(world_size, accumulation, batch) < 1:
+        raise ValueError("World size, accumulation and batch size must be positive")
+    configured_manifest = OmegaConf.select(cfg, "datasets.vla_data.valid_anchor_manifest_path")
+    manifest = args.manifest or (Path(configured_manifest) if configured_manifest else None)
+    prepare_manifest = manifest is None
+    if prepare_manifest and args.start == "resume":
+        raise ValueError("Exact resume requires the parent run's manifest; use --manifest")
+    manifest = output / "proposal" / "manifest.json" if prepare_manifest else _file(manifest, "manifest")
+    if configured_manifest and Path(configured_manifest).expanduser().resolve() != manifest:
+        raise ValueError("--manifest differs from the config manifest")
+    base = args.base_policy.expanduser().resolve()
+    if args.start == "resume":
+        checkpoint, _ = _validate_resume_checkpoint(base, world_size)
+    else:
+        checkpoint = _file(base / "pytorch_model.pt" if base.is_dir() else base, "base-policy model")
+    cfg.trainer.pretrained_checkpoint = str(checkpoint) if args.start == "weights" else None
+    cfg.trainer.resume_accelerate_checkpoint = str(base) if args.start == "resume" else None
+    cfg.trainer.is_resume = args.start == "resume"
+    modes = list(args.mode)
+    for mode in modes:
+        if (output / "runs" / mode).exists():
+            raise ValueError(f"Training output already exists: {output / 'runs' / mode}; choose a fresh --output")
+    commands = []
+    cfg.datasets.vla_data.valid_anchor_manifest_path = str(manifest)
+    # Materialize all paths once so every arm and subprocess uses the same config.
+    config_text = OmegaConf.to_yaml(cfg, resolve=True)
+    config = output / "config.yaml"
+    if prepare_manifest:
+        commands.append([sys.executable, "-m", "retime.prepare", "--config", str(config), "--output", str(manifest)])
+
+    def script(name: str, *arguments: str) -> None:
+        module = {
+            "build_residual_temporal_proposal.py": "retime.commands.proposal",
+            "build_shuffle_temporal_proposal.py": "retime.commands.shuffle",
+        }[name]
+        commands.append([sys.executable, "-m", module, *map(str, arguments)])
+
+    residual = output / "proposal" / "retime.json"
+    shuffled = output / "proposal" / "shuffle.json"
+    if any(mode != "uniform" for mode in modes):
+        script("build_residual_temporal_proposal.py", "--config", config, "--checkpoint", checkpoint,
+               "--manifest", manifest, "--cache-dir", output / "cache",
+               "--output", residual, "--grid-size", args.k, "--alpha", args.alpha, "--epsilon", args.epsilon)
+        if "shuffle" in modes:
+            script("build_shuffle_temporal_proposal.py", "--manifest", manifest,
+                   "--residual-proposal", residual, "--permutation-seed", args.shuffle_seed,
+                   "--output", shuffled)
+    training = _file(root / "starVLA" / "training" / "train_starvla.py", "training backend")
+    for mode in modes:
+        proposal = {"uniform": None, "retime": str(residual), "shuffle": str(shuffled)}[mode]
+        overrides = {
+            "run_id": mode,
+            "run_root_dir": str(output / "runs"),
+            "output_dir": str(output / "runs" / mode),
+            "datasets.vla_data.valid_anchor_manifest_path": str(manifest),
+            "datasets.vla_data.temporal_proposal_path": proposal,
+            "trainer.is_resume": args.start == "resume",
+            "trainer.allow_model_only_resume": False,
+            "trainer.resume_accelerate_checkpoint": str(base) if args.start == "resume" else None,
+            "trainer.pretrained_checkpoint": str(checkpoint) if args.start == "weights" else None,
+        }
+        commands.append([
+            sys.executable, "-m", "accelerate.commands.launch", "--config_file", str(launch_config),
+            "--num_processes", str(world_size), str(training), "--config_yaml", str(config),
+            *[f"--{key}={json.dumps(value)}" for key, value in overrides.items()],
+        ])
+    return {
+        "backend_root": str(root), "output": str(output), "base_policy": str(base),
+        "start": args.start, "alpha": args.alpha, "K": args.k, "modes": modes,
+        "global_batch_size": batch * world_size * accumulation,
+        "config_text": config_text,
+        "commands": commands,
+    }
+
+
+def main(argv: list[str] | None = None) -> int:
+    arguments = parser()
+    args = arguments.parse_args(argv)
+    try:
+        plan = build_plan(args)
+    except (ValueError, OSError, ImportError, KeyError) as error:
+        arguments.error(str(error))
+    print(json.dumps({key: value for key, value in plan.items() if key != "config_text"}, indent=2), flush=True)
+    if args.dry_run:
+        return 0
+    output = Path(plan["output"])
+    output.mkdir(parents=True, exist_ok=True)
+    (output / "proposal").mkdir(exist_ok=True)
+    (output / "config.yaml").write_text(plan["config_text"], encoding="utf-8")
+    (output / "plan.json").write_text(json.dumps(plan, indent=2) + "\n", encoding="utf-8")
+    env = os.environ.copy()
+    env["PYTHONPATH"] = os.pathsep.join(filter(None, [plan["backend_root"], env.get("PYTHONPATH")]))
+    env.setdefault("WANDB_MODE", "offline")
+    env["PYTHONUNBUFFERED"] = "1"
+    for command in plan["commands"]:
+        subprocess.run(command, cwd=plan["backend_root"], env=env, check=True)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

real_robot/source/retime/commands/__init__.py

@@ -0,0 +1 @@
+"""ReTime proposal-generation commands."""