Add EgoTouch T-ReX editor replica checkpoint and strict loader
Browse files- README.md +4 -0
- model/trex_editor_egotouch_replica_20260930/README.md +58 -0
- model/trex_editor_egotouch_replica_20260930/best.pt +3 -0
- model/trex_editor_egotouch_replica_20260930/config.json +56 -0
- model/trex_editor_egotouch_replica_20260930/load_editor.py +19 -0
- model/trex_editor_egotouch_replica_20260930/verification.json +34 -0
README.md
CHANGED
|
@@ -57,3 +57,7 @@ Its checkpoint uses `human_trex.full.v1`, not the legacy baseline format describ
|
|
| 57 |
[model/tachin_adaptation_20260922](model/tachin_adaptation_20260922) contains validation-selected SmolVLA, VITRA, and HumanTReX checkpoints after Tachin adaptation.
|
| 58 |
These are trained-parameter checkpoints, not standalone full models. The release documents their frozen initialization dependencies.
|
| 59 |
No tactile editor or full test results are included. The existing three-domain and EgoTouch releases remain unchanged.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
[model/tachin_adaptation_20260922](model/tachin_adaptation_20260922) contains validation-selected SmolVLA, VITRA, and HumanTReX checkpoints after Tachin adaptation.
|
| 58 |
These are trained-parameter checkpoints, not standalone full models. The release documents their frozen initialization dependencies.
|
| 59 |
No tactile editor or full test results are included. The existing three-domain and EgoTouch releases remain unchanged.
|
| 60 |
+
|
| 61 |
+
## EgoTouch T-ReX editor replica
|
| 62 |
+
|
| 63 |
+
[model/trex_editor_egotouch_replica_20260930](model/trex_editor_egotouch_replica_20260930) contains the validation-selected tactile editor from the September 30 video-generation replica. This is not the original paper-run checkpoint. It is separate from the full HumanTReX backbone. The package includes a strict loader and a verification record.
|
model/trex_editor_egotouch_replica_20260930/README.md
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# EgoTouch T-ReX editor — September 2026 replica
|
| 2 |
+
|
| 3 |
+
This package contains the external tactile editor for our no-touch HumanTReX backbone.
|
| 4 |
+
It is the September 30 replica for video generation, not the original paper-run checkpoint.
|
| 5 |
+
The replica does not replace the original reported results.
|
| 6 |
+
|
| 7 |
+
## Files
|
| 8 |
+
|
| 9 |
+
- `best.pt`: the validation-selected editor, including its trained tactile trunk and training statistics.
|
| 10 |
+
- `config.json`: the original run config. Local paths describe the original environment, not download requirements.
|
| 11 |
+
- `load_editor.py`: a strict loader for the public editor package.
|
| 12 |
+
- `verification.json`: checkpoint hash and CPU inference smoke results.
|
| 13 |
+
|
| 14 |
+
The selected checkpoint is from epoch 5, step 20,870. The run trained for five epochs.
|
| 15 |
+
The tactile CNN/GRU started from random initialization and trained with the editor.
|
| 16 |
+
This is not the P1 retrieval encoder or the W3 editor. It uses no memory tokens or world-model losses.
|
| 17 |
+
The backbone remained frozen through cached predictions.
|
| 18 |
+
|
| 19 |
+
## Dependencies
|
| 20 |
+
|
| 21 |
+
Backbone: [model/trex_full_egotouch](../trex_full_egotouch).
|
| 22 |
+
This editor package does not contain the backbone, datasets, or caches.
|
| 23 |
+
|
| 24 |
+
Editor implementation: [T-WLA-public/tactile_editor](https://github.com/kaichen-z/VLA-HAND/tree/d0731ac045298641e37251fd16cebd24e6f7b34c/tactile_editor).
|
| 25 |
+
Install that package with `pip install -e ./tactile_editor` from the repository root.
|
| 26 |
+
|
| 27 |
+
## Download and load
|
| 28 |
+
|
| 29 |
+
```python
|
| 30 |
+
from pathlib import Path
|
| 31 |
+
import sys
|
| 32 |
+
from huggingface_hub import snapshot_download
|
| 33 |
+
|
| 34 |
+
root = Path(snapshot_download(
|
| 35 |
+
"MIT-Media-Lab/vla-hand-baselines",
|
| 36 |
+
allow_patterns=["model/trex_editor_egotouch_replica_20260930/*"],
|
| 37 |
+
)) / "model/trex_editor_egotouch_replica_20260930"
|
| 38 |
+
sys.path.insert(0, str(root))
|
| 39 |
+
from load_editor import load_editor
|
| 40 |
+
editor, statistics = load_editor(root / "best.pt", device="cuda")
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
## Input contract
|
| 44 |
+
|
| 45 |
+
The editor accepts normalized base actions `[B,16,192]`, eight raw EgoTouch pressure frames `[B,8,2,21,21]`, and temporal validity `[B,8,2]`.
|
| 46 |
+
Additional inputs are the edit mask `[B,16,192]` and plan-relative decision time `[B]` in seconds.
|
| 47 |
+
The internal trunk applies the left-glove grid flip. Input histories must not apply that flip a second time.
|
| 48 |
+
|
| 49 |
+
Normalize raw actions with `statistics['action_mean']` and `statistics['action_std']` before the editor call.
|
| 50 |
+
Convert edited actions back to physical units with the same statistics.
|
| 51 |
+
The editor returns `(normalized_actions, details)`.
|
| 52 |
+
|
| 53 |
+
Offline replay uses 10 Hz plans, 80 Hz actions, and decisions at 0, 33.3, and 66.7 ms.
|
| 54 |
+
Each revision starts from the immutable base plan. The mask excludes committed actions and padding dimensions 138:192.
|
| 55 |
+
Evaluation executes the first eight actions with the latest available correction.
|
| 56 |
+
|
| 57 |
+
The original config notes upstream split exposure. These are development results, not evidence of a fully isolated benchmark.
|
| 58 |
+
The release smoke test checks strict loading, finite outputs, and unchanged masked actions. It is not a new full evaluation.
|
model/trex_editor_egotouch_replica_20260930/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0567384be68a055610620ae7bdad97fbedf8318a03dca24f54f14331995ae6fd
|
| 3 |
+
size 2461723
|
model/trex_editor_egotouch_replica_20260930/config.json
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"root": "/scratch/chonghej/VLA/video_regeneration_20260929/trex",
|
| 3 |
+
"variant": "trex",
|
| 4 |
+
"epochs": 5,
|
| 5 |
+
"micro_batch": 64,
|
| 6 |
+
"effective_batch": 256,
|
| 7 |
+
"workers": 4,
|
| 8 |
+
"seed": 20260921,
|
| 9 |
+
"smoke_steps": 0,
|
| 10 |
+
"resume": false,
|
| 11 |
+
"output_dir": "/scratch/chonghej/VLA/video_regeneration_20260929/trex/runs/trex",
|
| 12 |
+
"lambda_rgb": 0.0,
|
| 13 |
+
"lambda_tactile": 0.0,
|
| 14 |
+
"diagnostic_every": 0,
|
| 15 |
+
"smoke_full_weights": false,
|
| 16 |
+
"action_lr": 0.0001,
|
| 17 |
+
"encoder_lr": 0.0001,
|
| 18 |
+
"aux_delay_epochs": 0,
|
| 19 |
+
"aux_ramp_steps": 400,
|
| 20 |
+
"encoder_freeze_epochs": 0,
|
| 21 |
+
"roi_background": 0.1,
|
| 22 |
+
"wm_action_gradient": 0.0,
|
| 23 |
+
"validation_only": false,
|
| 24 |
+
"train_rows": 1068435,
|
| 25 |
+
"validation_rows": 60640,
|
| 26 |
+
"query_identity": {
|
| 27 |
+
"plans": 415624,
|
| 28 |
+
"edit_rows": 1068435,
|
| 29 |
+
"counts_by_tick": [
|
| 30 |
+
415624,
|
| 31 |
+
353877,
|
| 32 |
+
298934
|
| 33 |
+
],
|
| 34 |
+
"query_sha256": "e4f4fb8e3e552fb8bfc2fd2bd18b99ba0f776c90f869821e0a5e350639050307",
|
| 35 |
+
"decision_ticks": [
|
| 36 |
+
0,
|
| 37 |
+
1,
|
| 38 |
+
2
|
| 39 |
+
],
|
| 40 |
+
"action_hz": 80,
|
| 41 |
+
"replan_hz": 10,
|
| 42 |
+
"tactile_hz": 30,
|
| 43 |
+
"rgb_target_reference": "plan_start + [0.1,0.2] seconds",
|
| 44 |
+
"future_tactile_reference": "decision + 0.1 seconds",
|
| 45 |
+
"baseline_gt_max_difference": 0.0,
|
| 46 |
+
"baseline_mask_mismatch": 0,
|
| 47 |
+
"t0_touch_max_difference": 0.0,
|
| 48 |
+
"source_time_policy": "original source frames at 30 Hz; PCHIP pose labels at 80 Hz; no extrapolation; causal hold-last RGB for missing historical frames"
|
| 49 |
+
},
|
| 50 |
+
"action_scaling": "shared train-GT mean/std",
|
| 51 |
+
"world_weight_warmup": 400,
|
| 52 |
+
"seed_note": "same row shuffle and effective batch for runs with the same seed",
|
| 53 |
+
"test_exposure": "See upstream_split_audit.json; cross-backbone comparison remains a development result.",
|
| 54 |
+
"trainable_components": "standard editor including randomly initialized CNN/GRU tactile trunk; cached trex frozen; no memory tokens enter editor",
|
| 55 |
+
"pretrained_retrieval_encoder": false
|
| 56 |
+
}
|
model/trex_editor_egotouch_replica_20260930/load_editor.py
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Load the EgoTouch T-ReX editor replica with the public editor package."""
|
| 2 |
+
import torch
|
| 3 |
+
from tactile_editor.direct_residual import DirectTactileResidualEditor
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
def load_editor(checkpoint, device="cpu"):
|
| 7 |
+
# Load only checkpoints from a trusted source. This file contains metadata too.
|
| 8 |
+
saved = torch.load(checkpoint, map_location="cpu", weights_only=False)
|
| 9 |
+
if saved["config"]["variant"] != "trex":
|
| 10 |
+
raise ValueError("Expected a T-ReX editor checkpoint")
|
| 11 |
+
stats = saved["statistics"]
|
| 12 |
+
model = DirectTactileResidualEditor(
|
| 13 |
+
action_mean=torch.zeros(192), action_std=torch.ones(192),
|
| 14 |
+
pressure_masks=stats["pressure_masks"],
|
| 15 |
+
reference_pressure=stats["reference_pressure"],
|
| 16 |
+
mode="history", residual_bases=8, articulation_only=False,
|
| 17 |
+
)
|
| 18 |
+
model.load_state_dict(saved["model"], strict=True)
|
| 19 |
+
return model.to(device).eval(), stats
|
model/trex_editor_egotouch_replica_20260930/verification.json
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"status": "PASS",
|
| 3 |
+
"replica": true,
|
| 4 |
+
"original_paper_checkpoint": false,
|
| 5 |
+
"code_revision": "d0731ac045298641e37251fd16cebd24e6f7b34c",
|
| 6 |
+
"strict_load": true,
|
| 7 |
+
"synthetic_cpu_forward": true,
|
| 8 |
+
"finite_output": true,
|
| 9 |
+
"masked_actions_unchanged": true,
|
| 10 |
+
"shape": [
|
| 11 |
+
3,
|
| 12 |
+
16,
|
| 13 |
+
192
|
| 14 |
+
],
|
| 15 |
+
"full_evaluation_rerun": false,
|
| 16 |
+
"files": {
|
| 17 |
+
"best.pt": {
|
| 18 |
+
"bytes": 2461723,
|
| 19 |
+
"sha256": "0567384be68a055610620ae7bdad97fbedf8318a03dca24f54f14331995ae6fd"
|
| 20 |
+
},
|
| 21 |
+
"config.json": {
|
| 22 |
+
"bytes": 1878,
|
| 23 |
+
"sha256": "eee545aba960b75083ddbbda044e81fbe3103afd174145bc93976748f4548ae7"
|
| 24 |
+
},
|
| 25 |
+
"README.md": {
|
| 26 |
+
"bytes": 2977,
|
| 27 |
+
"sha256": "af8b9d6ffb8fcf51f484bf62478aca7c96d450a1736e92ddf84b5fdae65f5e87"
|
| 28 |
+
},
|
| 29 |
+
"load_editor.py": {
|
| 30 |
+
"bytes": 886,
|
| 31 |
+
"sha256": "1c27e83657550f1d7ac4af52e348a9006bd80ce2c107b2d88b37b5de70253c13"
|
| 32 |
+
}
|
| 33 |
+
}
|
| 34 |
+
}
|