maskjp commited on
Commit
63c8ac9
·
verified ·
1 Parent(s): 6db6e0f

mtdit_eef @ step 070000 (held-out-loss minimum)

Browse files
README.md ADDED
@@ -0,0 +1,77 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: lerobot
4
+ tags: [robotics, lerobot, mtdit-spatial, multi-task-dit, relative-actions]
5
+ ---
6
+
7
+ # MTDiT-Spatial — relative actions, end-effector space (13-dim pose, 6D rotation)
8
+
9
+ `mtdit_spatial` with `vision_feature_mode=cls`, which is byte-identical to stock
10
+ `multi_task_dit`. Trained **from scratch** on the 949-episode `base4` multi-task mixture
11
+ (900 train / 49 held out, 8 tasks, 3 cameras, 50 Hz).
12
+
13
+ Checkpoint **step 070000** of a 100K-step run — the held-out-loss minimum, not the final step
14
+ (earliest of a four-way tie at 0.0074 (70K/75K/85K/90K); fewest steps at the floor).
15
+
16
+ ## What this is for
17
+
18
+ One cell of a 2x2 matrix crossing action representation (relative joints vs relative
19
+ end-effector) with policy (MTDiT vs pi0.5), testing whether a relative action target makes the
20
+ policy attend to its cameras. **It does not.** These weights are published as a negative result.
21
+
22
+ | | value |
23
+ |---|---|
24
+ | held-out loss | 0.0074 |
25
+ | camera-sensitivity ratio | 0.119 |
26
+ | prediction error | 0.104 |
27
+ | null test | 0.00000 (pass) |
28
+
29
+ Measured on `multitask-eefabs6d-v30` episodes 93-97 -- the same croissant episodes, in end-effector space -- scored on position dims 0:3 (metres).
30
+
31
+ The ratio is **not comparable to the joint-space sibling**: this one divides a change in predicted end-effector position (metres) by the natural spread of that quantity, the other divides degrees by degrees. Only within-representation comparisons are meaningful, which is also why no GROUNDED/WEAK/BLIND word is given here -- those boundaries were calibrated on joint-space targets.
32
+
33
+ Camera sensitivity holds `observation.state` fixed, swaps in another episode's cameras from a
34
+ state-matched frame, and divides the resulting change in predicted arm joints by the natural
35
+ cross-episode spread of the ground truth. **1.0 means the policy tracks the scene; 0.0 means it
36
+ ignores it.** Measured on croissant held-out episodes 93-97 restricted to the "pick up the
37
+ croissant" phase, four inference seeds under common random numbers, padded steps excluded, with
38
+ the null test (each variant fed the anchor's own cameras) required to return 0.
39
+
40
+ Read the ratio together with the prediction error: a high ratio with a high error is an unstable
41
+ policy moving because any input moved, not a grounded one. The error here is well inside the 0.8
42
+ reliability gate.
43
+
44
+ ## Action representation
45
+
46
+ Targets are **relative**: `action[t+k] - state[anchor]`, one anchor per chunk, added back after
47
+ inference. The gripper stays absolute (`relative_exclude_joints=['gripper']`) because it is a
48
+ near-binary command with a 70-unit single-step jump, not a pose.
49
+
50
+ Normalisation statistics must match the representation, so this trains against `l5vel-peng/multitask-eefrel-h32`, a
51
+ metadata-only sibling of the mixture whose `meta/stats.json` holds offset quantiles computed at
52
+ the policy horizon. Training against absolute statistics would squash every target.
53
+
54
+ ## Configuration
55
+
56
+ ```
57
+ horizon 32 n_action_steps 24 n_obs_steps 2
58
+ hidden_dim 512 num_layers 4 num_heads 8 dropout 0.1
59
+ diffusion / DDPM, num_train_timesteps 100
60
+ optimizer_lr 3e-4 weight_decay 0 cosine to 100K, warmup 0
61
+ CLIP openai/clip-vit-base-patch16, vision_encoder_lr_multiplier 0.1
62
+ images resize [240,320] -> random crop [224,224]
63
+ norm VISUAL MEAN_STD | STATE MIN_MAX | ACTION QUANTILES
64
+ batch 64, bf16, seed 1000
65
+ ```
66
+
67
+ ## Loading
68
+
69
+ ```python
70
+ from lerobot.policies.factory import make_policy # needs lerobot_policy_mtdit_spatial installed
71
+ ```
72
+
73
+ Requires the [`lerobot_policy_mtdit_spatial`](https://github.com/) plugin, which registers the
74
+ `mtdit_spatial` policy type. Loading with `--policy.path` inherits this checkpoint's config;
75
+ `--policy.pretrained_path` loads weights only and resets stored settings to defaults.
76
+
77
+ Apache-2.0, inherited from LeRobot.
config.json ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "type": "mtdit_spatial",
3
+ "n_obs_steps": 2,
4
+ "input_features": {
5
+ "observation.state": {
6
+ "type": "STATE",
7
+ "shape": [
8
+ 13
9
+ ]
10
+ },
11
+ "observation.images.left": {
12
+ "type": "VISUAL",
13
+ "shape": [
14
+ 3,
15
+ 480,
16
+ 640
17
+ ]
18
+ },
19
+ "observation.images.right": {
20
+ "type": "VISUAL",
21
+ "shape": [
22
+ 3,
23
+ 480,
24
+ 640
25
+ ]
26
+ },
27
+ "observation.images.wrist": {
28
+ "type": "VISUAL",
29
+ "shape": [
30
+ 3,
31
+ 480,
32
+ 640
33
+ ]
34
+ },
35
+ "observation.base_pose": {
36
+ "type": "STATE",
37
+ "shape": [
38
+ 3
39
+ ]
40
+ },
41
+ "observation.eef_state": {
42
+ "type": "STATE",
43
+ "shape": [
44
+ 13
45
+ ]
46
+ }
47
+ },
48
+ "output_features": {
49
+ "action": {
50
+ "type": "ACTION",
51
+ "shape": [
52
+ 13
53
+ ]
54
+ },
55
+ "action.eef": {
56
+ "type": "ACTION",
57
+ "shape": [
58
+ 13
59
+ ]
60
+ }
61
+ },
62
+ "device": "cuda",
63
+ "use_amp": false,
64
+ "use_peft": false,
65
+ "push_to_hub": false,
66
+ "repo_id": null,
67
+ "private": null,
68
+ "tags": null,
69
+ "license": null,
70
+ "pretrained_path": null,
71
+ "pretrained_revision": null,
72
+ "horizon": 32,
73
+ "n_action_steps": 24,
74
+ "objective": "diffusion",
75
+ "noise_scheduler_type": "DDPM",
76
+ "num_train_timesteps": 100,
77
+ "beta_schedule": "squaredcos_cap_v2",
78
+ "beta_start": 0.0001,
79
+ "beta_end": 0.02,
80
+ "prediction_type": "epsilon",
81
+ "clip_sample": true,
82
+ "clip_sample_range": 1.0,
83
+ "num_inference_steps": null,
84
+ "sigma_min": 0.0,
85
+ "num_integration_steps": 100,
86
+ "integration_method": "euler",
87
+ "timestep_sampling_strategy": "beta",
88
+ "timestep_sampling_s": 0.999,
89
+ "timestep_sampling_alpha": 1.5,
90
+ "timestep_sampling_beta": 1.0,
91
+ "hidden_dim": 512,
92
+ "num_layers": 4,
93
+ "num_heads": 8,
94
+ "dropout": 0.1,
95
+ "use_positional_encoding": false,
96
+ "timestep_embed_dim": 256,
97
+ "use_rope": true,
98
+ "rope_base": 10000.0,
99
+ "vision_encoder_name": "openai/clip-vit-base-patch16",
100
+ "use_separate_rgb_encoder_per_camera": false,
101
+ "vision_encoder_lr_multiplier": 0.1,
102
+ "image_resize_shape": [
103
+ 240,
104
+ 320
105
+ ],
106
+ "image_crop_shape": [
107
+ 224,
108
+ 224
109
+ ],
110
+ "image_crop_is_random": true,
111
+ "vision_feature_mode": "cls",
112
+ "num_spatial_keypoints": 32,
113
+ "spatial_softmax_keep_cls": true,
114
+ "spatial_softmax_learnable_temperature": true,
115
+ "cross_attention_heads": null,
116
+ "action_stride": 1,
117
+ "state_dropout_prob": 0.0,
118
+ "action_loss_weighting": "none",
119
+ "loss_weight_max": 4.0,
120
+ "loss_gripper_dim": 9,
121
+ "state_conditioning": "full",
122
+ "state_feature_names": null,
123
+ "masked_state_names": null,
124
+ "use_relative_actions": true,
125
+ "relative_exclude_joints": [
126
+ "gripper"
127
+ ],
128
+ "action_feature_names": [
129
+ "eef_x",
130
+ "eef_y",
131
+ "eef_z",
132
+ "eef_xx",
133
+ "eef_xy",
134
+ "eef_xz",
135
+ "eef_yx",
136
+ "eef_yy",
137
+ "eef_yz",
138
+ "gripper",
139
+ "base_x",
140
+ "base_y",
141
+ "base_yaw"
142
+ ],
143
+ "text_encoder_name": "openai/clip-vit-base-patch16",
144
+ "tokenizer_max_length": 77,
145
+ "tokenizer_padding": "max_length",
146
+ "tokenizer_padding_side": "right",
147
+ "tokenizer_truncation": true,
148
+ "normalization_mapping": {
149
+ "VISUAL": "MEAN_STD",
150
+ "STATE": "MIN_MAX",
151
+ "ACTION": "QUANTILES"
152
+ },
153
+ "optimizer_lr": 0.0003,
154
+ "optimizer_betas": [
155
+ 0.95,
156
+ 0.999
157
+ ],
158
+ "optimizer_eps": 1e-08,
159
+ "optimizer_weight_decay": 0.0,
160
+ "scheduler_name": "cosine",
161
+ "scheduler_warmup_steps": 0,
162
+ "do_mask_loss_for_padding": false,
163
+ "drop_n_last_frames": 7
164
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bbfe1baf1250312473ad0cf7f977ea78a6704a986a0fa9e545ec4f27e6242845
3
+ size 939228980
policy_postprocessor.json ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "policy_postprocessor",
3
+ "steps": [
4
+ {
5
+ "registry_name": "unnormalizer_processor",
6
+ "config": {
7
+ "eps": 1e-08,
8
+ "features": {
9
+ "action": {
10
+ "type": "ACTION",
11
+ "shape": [
12
+ 13
13
+ ]
14
+ },
15
+ "action.eef": {
16
+ "type": "ACTION",
17
+ "shape": [
18
+ 13
19
+ ]
20
+ }
21
+ },
22
+ "norm_map": {
23
+ "VISUAL": "MEAN_STD",
24
+ "STATE": "MIN_MAX",
25
+ "ACTION": "QUANTILES"
26
+ }
27
+ },
28
+ "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
29
+ },
30
+ {
31
+ "registry_name": "absolute_actions_processor",
32
+ "config": {
33
+ "enabled": true
34
+ }
35
+ },
36
+ {
37
+ "registry_name": "device_processor",
38
+ "config": {
39
+ "device": "cpu",
40
+ "float_dtype": null
41
+ }
42
+ }
43
+ ]
44
+ }
policy_postprocessor_step_0_unnormalizer_processor.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3966986d9e5ad90d51efceb5f6542c330c01a99537d08b5b77f4beb9f91b874f
3
+ size 11624
policy_preprocessor.json ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "policy_preprocessor",
3
+ "steps": [
4
+ {
5
+ "registry_name": "rename_observations_processor",
6
+ "config": {
7
+ "rename_map": {}
8
+ }
9
+ },
10
+ {
11
+ "registry_name": "to_batch_processor",
12
+ "config": {}
13
+ },
14
+ {
15
+ "registry_name": "tokenizer_processor",
16
+ "config": {
17
+ "max_length": 77,
18
+ "task_key": "task",
19
+ "padding_side": "right",
20
+ "padding": "max_length",
21
+ "truncation": true,
22
+ "tokenizer_name": "tokenizer"
23
+ },
24
+ "artifacts": {
25
+ "tokenizer_name": "tokenizer"
26
+ }
27
+ },
28
+ {
29
+ "registry_name": "device_processor",
30
+ "config": {
31
+ "device": "cuda",
32
+ "float_dtype": null
33
+ }
34
+ },
35
+ {
36
+ "registry_name": "mtdit_spatial_relative_actions_processor",
37
+ "config": {
38
+ "enabled": true,
39
+ "exclude_joints": [
40
+ "gripper"
41
+ ],
42
+ "action_names": [
43
+ "eef_x",
44
+ "eef_y",
45
+ "eef_z",
46
+ "eef_xx",
47
+ "eef_xy",
48
+ "eef_xz",
49
+ "eef_yx",
50
+ "eef_yy",
51
+ "eef_yz",
52
+ "gripper",
53
+ "base_x",
54
+ "base_y",
55
+ "base_yaw"
56
+ ]
57
+ }
58
+ },
59
+ {
60
+ "registry_name": "normalizer_processor",
61
+ "config": {
62
+ "eps": 1e-08,
63
+ "features": {
64
+ "observation.state": {
65
+ "type": "STATE",
66
+ "shape": [
67
+ 13
68
+ ]
69
+ },
70
+ "observation.images.left": {
71
+ "type": "VISUAL",
72
+ "shape": [
73
+ 3,
74
+ 480,
75
+ 640
76
+ ]
77
+ },
78
+ "observation.images.right": {
79
+ "type": "VISUAL",
80
+ "shape": [
81
+ 3,
82
+ 480,
83
+ 640
84
+ ]
85
+ },
86
+ "observation.images.wrist": {
87
+ "type": "VISUAL",
88
+ "shape": [
89
+ 3,
90
+ 480,
91
+ 640
92
+ ]
93
+ },
94
+ "observation.base_pose": {
95
+ "type": "STATE",
96
+ "shape": [
97
+ 3
98
+ ]
99
+ },
100
+ "observation.eef_state": {
101
+ "type": "STATE",
102
+ "shape": [
103
+ 13
104
+ ]
105
+ },
106
+ "action": {
107
+ "type": "ACTION",
108
+ "shape": [
109
+ 13
110
+ ]
111
+ },
112
+ "action.eef": {
113
+ "type": "ACTION",
114
+ "shape": [
115
+ 13
116
+ ]
117
+ }
118
+ },
119
+ "norm_map": {
120
+ "VISUAL": "MEAN_STD",
121
+ "STATE": "MIN_MAX",
122
+ "ACTION": "QUANTILES"
123
+ }
124
+ },
125
+ "state_file": "policy_preprocessor_step_5_normalizer_processor.safetensors"
126
+ }
127
+ ]
128
+ }
policy_preprocessor_step_5_normalizer_processor.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:20d2a04d3d4c0b1508c5898b0e446b95d798615864f0aee891b0bceb0ff8ed00
3
+ size 11640
tokenizer/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer/tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|startoftext|>",
5
+ "do_lower_case": true,
6
+ "eos_token": "<|endoftext|>",
7
+ "errors": "replace",
8
+ "is_local": false,
9
+ "model_max_length": 77,
10
+ "pad_token": "<|endoftext|>",
11
+ "tokenizer_class": "CLIPTokenizer",
12
+ "unk_token": "<|endoftext|>"
13
+ }
train_config.json ADDED
@@ -0,0 +1,365 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "dataset": {
3
+ "repo_id": "l5vel-peng/multitask-eefrel-h32",
4
+ "repo_type": "dataset",
5
+ "root": "/home/data/pej3435/.cache/l5vel_eef_merge/outputs/multitask-eefrel-h32",
6
+ "episodes": null,
7
+ "exclude_episodes": null,
8
+ "image_transforms": {
9
+ "enable": false,
10
+ "max_num_transforms": 3,
11
+ "random_order": false,
12
+ "tfs": {
13
+ "brightness": {
14
+ "weight": 1.0,
15
+ "type": "ColorJitter",
16
+ "kwargs": {
17
+ "brightness": [
18
+ 0.8,
19
+ 1.2
20
+ ]
21
+ }
22
+ },
23
+ "contrast": {
24
+ "weight": 1.0,
25
+ "type": "ColorJitter",
26
+ "kwargs": {
27
+ "contrast": [
28
+ 0.8,
29
+ 1.2
30
+ ]
31
+ }
32
+ },
33
+ "saturation": {
34
+ "weight": 1.0,
35
+ "type": "ColorJitter",
36
+ "kwargs": {
37
+ "saturation": [
38
+ 0.5,
39
+ 1.5
40
+ ]
41
+ }
42
+ },
43
+ "hue": {
44
+ "weight": 1.0,
45
+ "type": "ColorJitter",
46
+ "kwargs": {
47
+ "hue": [
48
+ -0.05,
49
+ 0.05
50
+ ]
51
+ }
52
+ },
53
+ "sharpness": {
54
+ "weight": 1.0,
55
+ "type": "SharpnessJitter",
56
+ "kwargs": {
57
+ "sharpness": [
58
+ 0.5,
59
+ 1.5
60
+ ]
61
+ }
62
+ },
63
+ "affine": {
64
+ "weight": 1.0,
65
+ "type": "RandomAffine",
66
+ "kwargs": {
67
+ "degrees": [
68
+ -5.0,
69
+ 5.0
70
+ ],
71
+ "translate": [
72
+ 0.05,
73
+ 0.05
74
+ ]
75
+ }
76
+ }
77
+ }
78
+ },
79
+ "revision": null,
80
+ "use_imagenet_stats": true,
81
+ "video_backend": "torchcodec",
82
+ "return_uint8": false,
83
+ "depth_output_unit": "mm",
84
+ "streaming": false,
85
+ "eval_split": 0.05
86
+ },
87
+ "env": null,
88
+ "policy": {
89
+ "type": "mtdit_spatial",
90
+ "n_obs_steps": 2,
91
+ "input_features": {
92
+ "observation.state": {
93
+ "type": "STATE",
94
+ "shape": [
95
+ 13
96
+ ]
97
+ },
98
+ "observation.images.left": {
99
+ "type": "VISUAL",
100
+ "shape": [
101
+ 3,
102
+ 480,
103
+ 640
104
+ ]
105
+ },
106
+ "observation.images.right": {
107
+ "type": "VISUAL",
108
+ "shape": [
109
+ 3,
110
+ 480,
111
+ 640
112
+ ]
113
+ },
114
+ "observation.images.wrist": {
115
+ "type": "VISUAL",
116
+ "shape": [
117
+ 3,
118
+ 480,
119
+ 640
120
+ ]
121
+ },
122
+ "observation.base_pose": {
123
+ "type": "STATE",
124
+ "shape": [
125
+ 3
126
+ ]
127
+ },
128
+ "observation.eef_state": {
129
+ "type": "STATE",
130
+ "shape": [
131
+ 13
132
+ ]
133
+ }
134
+ },
135
+ "output_features": {
136
+ "action": {
137
+ "type": "ACTION",
138
+ "shape": [
139
+ 13
140
+ ]
141
+ },
142
+ "action.eef": {
143
+ "type": "ACTION",
144
+ "shape": [
145
+ 13
146
+ ]
147
+ }
148
+ },
149
+ "device": "cuda",
150
+ "use_amp": false,
151
+ "use_peft": false,
152
+ "push_to_hub": false,
153
+ "repo_id": null,
154
+ "private": null,
155
+ "tags": null,
156
+ "license": null,
157
+ "pretrained_path": null,
158
+ "pretrained_revision": null,
159
+ "horizon": 32,
160
+ "n_action_steps": 24,
161
+ "objective": "diffusion",
162
+ "noise_scheduler_type": "DDPM",
163
+ "num_train_timesteps": 100,
164
+ "beta_schedule": "squaredcos_cap_v2",
165
+ "beta_start": 0.0001,
166
+ "beta_end": 0.02,
167
+ "prediction_type": "epsilon",
168
+ "clip_sample": true,
169
+ "clip_sample_range": 1.0,
170
+ "num_inference_steps": null,
171
+ "sigma_min": 0.0,
172
+ "num_integration_steps": 100,
173
+ "integration_method": "euler",
174
+ "timestep_sampling_strategy": "beta",
175
+ "timestep_sampling_s": 0.999,
176
+ "timestep_sampling_alpha": 1.5,
177
+ "timestep_sampling_beta": 1.0,
178
+ "hidden_dim": 512,
179
+ "num_layers": 4,
180
+ "num_heads": 8,
181
+ "dropout": 0.1,
182
+ "use_positional_encoding": false,
183
+ "timestep_embed_dim": 256,
184
+ "use_rope": true,
185
+ "rope_base": 10000.0,
186
+ "vision_encoder_name": "openai/clip-vit-base-patch16",
187
+ "use_separate_rgb_encoder_per_camera": false,
188
+ "vision_encoder_lr_multiplier": 0.1,
189
+ "image_resize_shape": [
190
+ 240,
191
+ 320
192
+ ],
193
+ "image_crop_shape": [
194
+ 224,
195
+ 224
196
+ ],
197
+ "image_crop_is_random": true,
198
+ "vision_feature_mode": "cls",
199
+ "num_spatial_keypoints": 32,
200
+ "spatial_softmax_keep_cls": true,
201
+ "spatial_softmax_learnable_temperature": true,
202
+ "cross_attention_heads": null,
203
+ "action_stride": 1,
204
+ "state_dropout_prob": 0.0,
205
+ "action_loss_weighting": "none",
206
+ "loss_weight_max": 4.0,
207
+ "loss_gripper_dim": 9,
208
+ "state_conditioning": "full",
209
+ "state_feature_names": null,
210
+ "masked_state_names": null,
211
+ "use_relative_actions": true,
212
+ "relative_exclude_joints": [
213
+ "gripper"
214
+ ],
215
+ "action_feature_names": [
216
+ "eef_x",
217
+ "eef_y",
218
+ "eef_z",
219
+ "eef_xx",
220
+ "eef_xy",
221
+ "eef_xz",
222
+ "eef_yx",
223
+ "eef_yy",
224
+ "eef_yz",
225
+ "gripper",
226
+ "base_x",
227
+ "base_y",
228
+ "base_yaw"
229
+ ],
230
+ "text_encoder_name": "openai/clip-vit-base-patch16",
231
+ "tokenizer_max_length": 77,
232
+ "tokenizer_padding": "max_length",
233
+ "tokenizer_padding_side": "right",
234
+ "tokenizer_truncation": true,
235
+ "normalization_mapping": {
236
+ "VISUAL": "MEAN_STD",
237
+ "STATE": "MIN_MAX",
238
+ "ACTION": "QUANTILES"
239
+ },
240
+ "optimizer_lr": 0.0003,
241
+ "optimizer_betas": [
242
+ 0.95,
243
+ 0.999
244
+ ],
245
+ "optimizer_eps": 1e-08,
246
+ "optimizer_weight_decay": 0.0,
247
+ "scheduler_name": "cosine",
248
+ "scheduler_warmup_steps": 0,
249
+ "do_mask_loss_for_padding": false,
250
+ "drop_n_last_frames": 7
251
+ },
252
+ "reward_model": null,
253
+ "output_dir": "/home/data/pej3435/Code/dev/lerobot_policy_mtdit_spatial/outputs/relrep100k/mtdit_eef",
254
+ "job_name": "relrep100k_mtdit_eef",
255
+ "resume": false,
256
+ "seed": 1000,
257
+ "cudnn_deterministic": false,
258
+ "num_workers": 12,
259
+ "batch_size": 64,
260
+ "prefetch_factor": 4,
261
+ "persistent_workers": true,
262
+ "dataloader_multiprocessing_context": "spawn",
263
+ "steps": 100000,
264
+ "env_eval_freq": 0,
265
+ "log_freq": 200,
266
+ "eval_steps": 5000,
267
+ "max_eval_samples": 0,
268
+ "tolerance_s": 0.0001,
269
+ "save_checkpoint": true,
270
+ "save_freq": 10000,
271
+ "checkpoint_format": "safetensors",
272
+ "use_policy_training_preset": true,
273
+ "optimizer": {
274
+ "type": "adam",
275
+ "lr": 0.0003,
276
+ "weight_decay": 0.0,
277
+ "grad_clip_norm": 10.0,
278
+ "betas": [
279
+ 0.95,
280
+ 0.999
281
+ ],
282
+ "eps": 1e-08
283
+ },
284
+ "scheduler": {
285
+ "type": "diffuser",
286
+ "num_warmup_steps": 0,
287
+ "name": "cosine"
288
+ },
289
+ "parallelism": {
290
+ "dp_replicate": 1,
291
+ "dp_shard": 1,
292
+ "context_parallel": {
293
+ "ring_degree": 1,
294
+ "ulysses_degree": 1
295
+ },
296
+ "cfg_parallel": 1
297
+ },
298
+ "accelerator": {
299
+ "mixed_precision": "bf16",
300
+ "gradient_accumulation": {
301
+ "steps": 1
302
+ },
303
+ "fsdp": {
304
+ "reshard_after_forward": true,
305
+ "wrap_modules": null,
306
+ "min_num_params": null,
307
+ "cpu_offload": false,
308
+ "ignored_modules": null
309
+ },
310
+ "ddp": {
311
+ "find_unused_parameters": true,
312
+ "gradient_as_bucket_view": false,
313
+ "static_graph": false
314
+ },
315
+ "compile": {
316
+ "enabled": false,
317
+ "backend": "inductor",
318
+ "mode": null,
319
+ "regional": true
320
+ },
321
+ "activation_checkpointing": {
322
+ "mode": "none"
323
+ }
324
+ },
325
+ "eval": {
326
+ "n_episodes": 50,
327
+ "batch_size": 50,
328
+ "use_async_envs": true,
329
+ "recording": false,
330
+ "recording_repo_id": null,
331
+ "recording_private": false
332
+ },
333
+ "ema": {
334
+ "enable": false,
335
+ "decay": null,
336
+ "update_after_step": 0,
337
+ "inv_gamma": 1.0,
338
+ "power": 0.75,
339
+ "min_decay": 0.0,
340
+ "max_decay": 0.9999,
341
+ "use_for_eval": true
342
+ },
343
+ "wandb": {
344
+ "enable": true,
345
+ "disable_artifact": true,
346
+ "project": "relative-representation-100k",
347
+ "entity": null,
348
+ "notes": null,
349
+ "run_id": "1iqc885k",
350
+ "mode": null,
351
+ "add_tags": true
352
+ },
353
+ "peft": null,
354
+ "job": {
355
+ "target": null,
356
+ "image": "huggingface/lerobot-gpu:latest",
357
+ "timeout": "2d",
358
+ "detach": false,
359
+ "tags": []
360
+ },
361
+ "save_checkpoint_to_hub": false,
362
+ "sample_weighting": null,
363
+ "rename_map": {},
364
+ "checkpoint_path": null
365
+ }