LeoJiangOR commited on
Commit
41a42fc
·
verified ·
1 Parent(s): 4722fb9

Add EgoTouch T-ReX editor replica checkpoint and strict loader

Browse files
README.md CHANGED
@@ -57,3 +57,7 @@ Its checkpoint uses `human_trex.full.v1`, not the legacy baseline format describ
57
  [model/tachin_adaptation_20260922](model/tachin_adaptation_20260922) contains validation-selected SmolVLA, VITRA, and HumanTReX checkpoints after Tachin adaptation.
58
  These are trained-parameter checkpoints, not standalone full models. The release documents their frozen initialization dependencies.
59
  No tactile editor or full test results are included. The existing three-domain and EgoTouch releases remain unchanged.
 
 
 
 
 
57
  [model/tachin_adaptation_20260922](model/tachin_adaptation_20260922) contains validation-selected SmolVLA, VITRA, and HumanTReX checkpoints after Tachin adaptation.
58
  These are trained-parameter checkpoints, not standalone full models. The release documents their frozen initialization dependencies.
59
  No tactile editor or full test results are included. The existing three-domain and EgoTouch releases remain unchanged.
60
+
61
+ ## EgoTouch T-ReX editor replica
62
+
63
+ [model/trex_editor_egotouch_replica_20260930](model/trex_editor_egotouch_replica_20260930) contains the validation-selected tactile editor from the September 30 video-generation replica. This is not the original paper-run checkpoint. It is separate from the full HumanTReX backbone. The package includes a strict loader and a verification record.
model/trex_editor_egotouch_replica_20260930/README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # EgoTouch T-ReX editor — September 2026 replica
2
+
3
+ This package contains the external tactile editor for our no-touch HumanTReX backbone.
4
+ It is the September 30 replica for video generation, not the original paper-run checkpoint.
5
+ The replica does not replace the original reported results.
6
+
7
+ ## Files
8
+
9
+ - `best.pt`: the validation-selected editor, including its trained tactile trunk and training statistics.
10
+ - `config.json`: the original run config. Local paths describe the original environment, not download requirements.
11
+ - `load_editor.py`: a strict loader for the public editor package.
12
+ - `verification.json`: checkpoint hash and CPU inference smoke results.
13
+
14
+ The selected checkpoint is from epoch 5, step 20,870. The run trained for five epochs.
15
+ The tactile CNN/GRU started from random initialization and trained with the editor.
16
+ This is not the P1 retrieval encoder or the W3 editor. It uses no memory tokens or world-model losses.
17
+ The backbone remained frozen through cached predictions.
18
+
19
+ ## Dependencies
20
+
21
+ Backbone: [model/trex_full_egotouch](../trex_full_egotouch).
22
+ This editor package does not contain the backbone, datasets, or caches.
23
+
24
+ Editor implementation: [T-WLA-public/tactile_editor](https://github.com/kaichen-z/VLA-HAND/tree/d0731ac045298641e37251fd16cebd24e6f7b34c/tactile_editor).
25
+ Install that package with `pip install -e ./tactile_editor` from the repository root.
26
+
27
+ ## Download and load
28
+
29
+ ```python
30
+ from pathlib import Path
31
+ import sys
32
+ from huggingface_hub import snapshot_download
33
+
34
+ root = Path(snapshot_download(
35
+ "MIT-Media-Lab/vla-hand-baselines",
36
+ allow_patterns=["model/trex_editor_egotouch_replica_20260930/*"],
37
+ )) / "model/trex_editor_egotouch_replica_20260930"
38
+ sys.path.insert(0, str(root))
39
+ from load_editor import load_editor
40
+ editor, statistics = load_editor(root / "best.pt", device="cuda")
41
+ ```
42
+
43
+ ## Input contract
44
+
45
+ The editor accepts normalized base actions `[B,16,192]`, eight raw EgoTouch pressure frames `[B,8,2,21,21]`, and temporal validity `[B,8,2]`.
46
+ Additional inputs are the edit mask `[B,16,192]` and plan-relative decision time `[B]` in seconds.
47
+ The internal trunk applies the left-glove grid flip. Input histories must not apply that flip a second time.
48
+
49
+ Normalize raw actions with `statistics['action_mean']` and `statistics['action_std']` before the editor call.
50
+ Convert edited actions back to physical units with the same statistics.
51
+ The editor returns `(normalized_actions, details)`.
52
+
53
+ Offline replay uses 10 Hz plans, 80 Hz actions, and decisions at 0, 33.3, and 66.7 ms.
54
+ Each revision starts from the immutable base plan. The mask excludes committed actions and padding dimensions 138:192.
55
+ Evaluation executes the first eight actions with the latest available correction.
56
+
57
+ The original config notes upstream split exposure. These are development results, not evidence of a fully isolated benchmark.
58
+ The release smoke test checks strict loading, finite outputs, and unchanged masked actions. It is not a new full evaluation.
model/trex_editor_egotouch_replica_20260930/best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0567384be68a055610620ae7bdad97fbedf8318a03dca24f54f14331995ae6fd
3
+ size 2461723
model/trex_editor_egotouch_replica_20260930/config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "root": "/scratch/chonghej/VLA/video_regeneration_20260929/trex",
3
+ "variant": "trex",
4
+ "epochs": 5,
5
+ "micro_batch": 64,
6
+ "effective_batch": 256,
7
+ "workers": 4,
8
+ "seed": 20260921,
9
+ "smoke_steps": 0,
10
+ "resume": false,
11
+ "output_dir": "/scratch/chonghej/VLA/video_regeneration_20260929/trex/runs/trex",
12
+ "lambda_rgb": 0.0,
13
+ "lambda_tactile": 0.0,
14
+ "diagnostic_every": 0,
15
+ "smoke_full_weights": false,
16
+ "action_lr": 0.0001,
17
+ "encoder_lr": 0.0001,
18
+ "aux_delay_epochs": 0,
19
+ "aux_ramp_steps": 400,
20
+ "encoder_freeze_epochs": 0,
21
+ "roi_background": 0.1,
22
+ "wm_action_gradient": 0.0,
23
+ "validation_only": false,
24
+ "train_rows": 1068435,
25
+ "validation_rows": 60640,
26
+ "query_identity": {
27
+ "plans": 415624,
28
+ "edit_rows": 1068435,
29
+ "counts_by_tick": [
30
+ 415624,
31
+ 353877,
32
+ 298934
33
+ ],
34
+ "query_sha256": "e4f4fb8e3e552fb8bfc2fd2bd18b99ba0f776c90f869821e0a5e350639050307",
35
+ "decision_ticks": [
36
+ 0,
37
+ 1,
38
+ 2
39
+ ],
40
+ "action_hz": 80,
41
+ "replan_hz": 10,
42
+ "tactile_hz": 30,
43
+ "rgb_target_reference": "plan_start + [0.1,0.2] seconds",
44
+ "future_tactile_reference": "decision + 0.1 seconds",
45
+ "baseline_gt_max_difference": 0.0,
46
+ "baseline_mask_mismatch": 0,
47
+ "t0_touch_max_difference": 0.0,
48
+ "source_time_policy": "original source frames at 30 Hz; PCHIP pose labels at 80 Hz; no extrapolation; causal hold-last RGB for missing historical frames"
49
+ },
50
+ "action_scaling": "shared train-GT mean/std",
51
+ "world_weight_warmup": 400,
52
+ "seed_note": "same row shuffle and effective batch for runs with the same seed",
53
+ "test_exposure": "See upstream_split_audit.json; cross-backbone comparison remains a development result.",
54
+ "trainable_components": "standard editor including randomly initialized CNN/GRU tactile trunk; cached trex frozen; no memory tokens enter editor",
55
+ "pretrained_retrieval_encoder": false
56
+ }
model/trex_editor_egotouch_replica_20260930/load_editor.py ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Load the EgoTouch T-ReX editor replica with the public editor package."""
2
+ import torch
3
+ from tactile_editor.direct_residual import DirectTactileResidualEditor
4
+
5
+
6
+ def load_editor(checkpoint, device="cpu"):
7
+ # Load only checkpoints from a trusted source. This file contains metadata too.
8
+ saved = torch.load(checkpoint, map_location="cpu", weights_only=False)
9
+ if saved["config"]["variant"] != "trex":
10
+ raise ValueError("Expected a T-ReX editor checkpoint")
11
+ stats = saved["statistics"]
12
+ model = DirectTactileResidualEditor(
13
+ action_mean=torch.zeros(192), action_std=torch.ones(192),
14
+ pressure_masks=stats["pressure_masks"],
15
+ reference_pressure=stats["reference_pressure"],
16
+ mode="history", residual_bases=8, articulation_only=False,
17
+ )
18
+ model.load_state_dict(saved["model"], strict=True)
19
+ return model.to(device).eval(), stats
model/trex_editor_egotouch_replica_20260930/verification.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "status": "PASS",
3
+ "replica": true,
4
+ "original_paper_checkpoint": false,
5
+ "code_revision": "d0731ac045298641e37251fd16cebd24e6f7b34c",
6
+ "strict_load": true,
7
+ "synthetic_cpu_forward": true,
8
+ "finite_output": true,
9
+ "masked_actions_unchanged": true,
10
+ "shape": [
11
+ 3,
12
+ 16,
13
+ 192
14
+ ],
15
+ "full_evaluation_rerun": false,
16
+ "files": {
17
+ "best.pt": {
18
+ "bytes": 2461723,
19
+ "sha256": "0567384be68a055610620ae7bdad97fbedf8318a03dca24f54f14331995ae6fd"
20
+ },
21
+ "config.json": {
22
+ "bytes": 1878,
23
+ "sha256": "eee545aba960b75083ddbbda044e81fbe3103afd174145bc93976748f4548ae7"
24
+ },
25
+ "README.md": {
26
+ "bytes": 2977,
27
+ "sha256": "af8b9d6ffb8fcf51f484bf62478aca7c96d450a1736e92ddf84b5fdae65f5e87"
28
+ },
29
+ "load_editor.py": {
30
+ "bytes": 886,
31
+ "sha256": "1c27e83657550f1d7ac4af52e348a9006bd80ce2c107b2d88b37b5de70253c13"
32
+ }
33
+ }
34
+ }