HaomingSong commited on
Commit
160b889
·
verified ·
1 Parent(s): f42750c

Upload cosmos3-edge base checkpoint in LeRobot format

Browse files

Converted from the NVIDIA release using LeRobot b8f060837. Includes configuration, processors, tokenizer, model card and license notices.

.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ OpenMDW License Agreement, version 1.1 (OpenMDW-1.1)
2
+
3
+ By exercising rights granted to you under this agreement, you accept and agree
4
+ to its terms.
5
+
6
+ As used in this agreement, "Model Materials" means the materials provided to
7
+ you under this agreement, consisting of: (1) one or more machine learning
8
+ models (including architecture and parameters); and (2) all related artifacts
9
+ (including associated data, documentation and software) that are provided to
10
+ you hereunder.
11
+
12
+ Subject to your compliance with this agreement, permission is hereby granted,
13
+ free of charge, to deal in the Model Materials without restriction, including
14
+ under all copyright, patent, database, and trade secret rights included or
15
+ embodied therein.
16
+
17
+ If you distribute any portion of the Model Materials, you shall retain in your
18
+ distribution (1) a copy of this agreement, and (2) all copyright notices and
19
+ other notices of origin included in the Model Materials that are applicable to
20
+ your distribution.
21
+
22
+ If you file, maintain, or voluntarily participate in a lawsuit against any
23
+ person or entity asserting that the Model Materials directly or indirectly
24
+ infringe any patent or copyright, then all rights and grants made to you
25
+ hereunder are terminated, unless that lawsuit was in response to a
26
+ corresponding lawsuit first brought against you.
27
+
28
+ This agreement does not impose any restrictions or obligations with respect to
29
+ any use, modification, or sharing of any outputs generated by using the Model
30
+ Materials.
31
+
32
+ THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
33
+ OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
34
+ FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE
35
+ ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL
36
+ TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
37
+
38
+ YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY
39
+ APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION
40
+ ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL
41
+ MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS
42
+ REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE
43
+ DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR
44
+ ANYTHING INCORPORATED OR EMBODIED THEREIN.
45
+
46
+ IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM,
47
+ DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
48
+ OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE
49
+ USE THEREOF OR OTHER DEALINGS THEREIN.
NOTICE ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Cosmos3-Edge base checkpoint in LeRobot format
2
+
3
+ Original model developer: NVIDIA.
4
+ Original model family: NVIDIA Cosmos(TM).
5
+ Original model repository: nvidia/Cosmos3-Edge
6
+ Exact original revision: 344d602b128d1bbdacb43b08d0a3626f46343e29
7
+ Original release: https://huggingface.co/nvidia/Cosmos3-Edge/tree/344d602b128d1bbdacb43b08d0a3626f46343e29
8
+ Original model card: https://huggingface.co/nvidia/Cosmos3-Edge/blob/344d602b128d1bbdacb43b08d0a3626f46343e29/README.md
9
+ Model license: OpenMDW License Agreement, version 1.1 (OpenMDW-1.1).
10
+
11
+ This distribution converts the NVIDIA release into the LeRobot robot-policy
12
+ checkpoint layout. It retains the modules required by that policy interface,
13
+ adds LeRobot configuration and serialized processors, and does not add
14
+ task-specific fine-tuning. The full native Cosmos inference API is not provided.
15
+
16
+ Conversion date: 2026-10-28 UTC.
17
+ LeRobot source revision: b8f060837ced0a4ce4b136ea65b4d56bc7894bed.
18
+ Conversion script SHA256:
19
+ a8660731d95e98c742ca3f813a493b8be12271e404364ee64d1d3bfc9468aa35
20
+
21
+ The OpenMDW 1.1 text is included without modification as LICENSE. Its source is
22
+ the official NVIDIA Cosmos repository at:
23
+ https://github.com/NVIDIA/cosmos/blob/c3e4354c96cc8c9c311babc93b51edc6fea43fb9/LICENSE
24
+
25
+ Runtime software is installed separately. The licenses of those dependencies
26
+ continue to apply to their own materials.
README.md ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: lerobot
3
+ pipeline_tag: robotics
4
+ license: other
5
+ license_name: openmdw1.1-license
6
+ license_link: https://openmdw.ai/license/1-1/
7
+ base_model: nvidia/Cosmos3-Edge
8
+ tags:
9
+ - lerobot
10
+ - cosmos3
11
+ - robotics
12
+ - pretrained
13
+ ---
14
+
15
+ # Cosmos3-Edge base for LeRobot
16
+
17
+ This repository contains the LeRobot policy conversion of the NVIDIA Cosmos3-Edge base checkpoint. It is a starting point for robot policy post-training. The conversion adds no task-specific fine-tuning.
18
+
19
+ The policy jointly predicts robot action chunks and future video latents from language and current camera observations, with optional current-state conditioning. It contains the model components needed by the LeRobot policy interface.
20
+
21
+ | Item | Value |
22
+ |---|---|
23
+ | Original model developer | NVIDIA |
24
+ | Original checkpoint | [nvidia/Cosmos3-Edge](https://huggingface.co/nvidia/Cosmos3-Edge/tree/344d602b128d1bbdacb43b08d0a3626f46343e29) |
25
+ | Exact source revision | `344d602b128d1bbdacb43b08d0a3626f46343e29` |
26
+ | LeRobot compatibility baseline | `b8f060837ced0a4ce4b136ea65b4d56bc7894bed` |
27
+ | Policy type | `cosmos3_edge` |
28
+ | Backbone | Nemotron 3 Dense VL MoT |
29
+ | Decoder layers / hidden width | 28 / 2048 |
30
+ | Stored scalar weights, including frozen components and VAE | 4,563,688,396 |
31
+ | Stored tensors | 1188, comprising 1184 BF16 and 4 FP32 |
32
+
33
+ The stored scalar count is not a count of trainable or active parameters. The package includes a Wan2.2 VAE, tokenizer files, and serialized LeRobot processors.
34
+
35
+ ## Loading and post-training
36
+
37
+ Use a LeRobot version containing the Cosmos3 policy types. Conversion and CPU loading checks used the exact commit above.
38
+
39
+ The saved observation and action features are empty. Frame rate and camera size are unresolved. A target dataset must provide these values before constructing the policy. Read the configuration with:
40
+
41
+ ```python
42
+ from lerobot.configs.policies import PreTrainedConfig
43
+
44
+ config = PreTrainedConfig.from_pretrained("HaomingSong/cosmos3-edge")
45
+ ```
46
+
47
+ The training entry points select different configurations:
48
+
49
+ - `--policy.path=HaomingSong/cosmos3-edge` preserves the saved DROID post-training recipe, including domain 8, 32-step chunks, state conditioning, identity normalization, UniPC with 4 inference steps, and guidance 3. State and action must already use the same layout and width.
50
+ - `--policy.type=cosmos3_edge --policy.pretrained_path=HaomingSong/cosmos3-edge` uses these weights with the size class's default LIBERO recipe, including domain 5, 16-step chunks, no state conditioning, action quantile normalization, UniPC with 30 steps, and guidance 1. The dataset must match that recipe's camera and action layout.
51
+
52
+ `load_action_modules` defaults to `true`, preserving the released action modules. For native-style post-training, `--policy.load_action_modules=false` reinitializes the action input projection, action output projection, and action modality embedding. The setting returns to `true` after loading so subsequent reloads preserve trained action modules.
53
+
54
+ The saved `dtype=float32` supports FP32 master weights during training. Most stored weights retain upstream BF16 precision. The four time-embedder tensors are FP32. For inference set `--policy.dtype=bfloat16`. For evaluation also set `--tf32=false`.
55
+
56
+ ## Scope and evaluation
57
+
58
+ The LeRobot implementation supports joint video-and-action `wam` mode, a single observation step, and the UniPC sampler. Actual action width comes from the target dataset. This package does not expose the upstream reasoning, audio, or standalone video-generation APIs.
59
+
60
+ CPU checks verified conversion and strict loading with resolved dataset features. No LIBERO closed-loop success rate is reported for this base conversion. Results for separately fine-tuned LIBERO checkpoints or NVIDIA Policy-DROID models do not measure this checkpoint.
61
+
62
+ ## License and provenance
63
+
64
+ The NVIDIA model materials use [OpenMDW 1.1](LICENSE). Retain the license and applicable copyright and origin notices when redistributing them. [NOTICE](NOTICE) records the original model and conversion attribution. Separately installed runtime software retains its own licenses.
65
+
66
+ Consult the original [NVIDIA model card](https://huggingface.co/nvidia/Cosmos3-Edge/blob/344d602b128d1bbdacb43b08d0a3626f46343e29/README.md) and its [safety](https://huggingface.co/nvidia/Cosmos3-Edge/blob/344d602b128d1bbdacb43b08d0a3626f46343e29/SAFETY.md), [bias](https://huggingface.co/nvidia/Cosmos3-Edge/blob/344d602b128d1bbdacb43b08d0a3626f46343e29/BIAS.md), [privacy](https://huggingface.co/nvidia/Cosmos3-Edge/blob/344d602b128d1bbdacb43b08d0a3626f46343e29/PRIVACY.md), and [explainability](https://huggingface.co/nvidia/Cosmos3-Edge/blob/344d602b128d1bbdacb43b08d0a3626f46343e29/EXPLAINABILITY.md) notes for upstream limitations.
config.json ADDED
@@ -0,0 +1,332 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "type": "cosmos3_edge",
3
+ "n_obs_steps": 1,
4
+ "input_features": {},
5
+ "output_features": {},
6
+ "device": "cuda",
7
+ "dtype": "float32",
8
+ "use_amp": false,
9
+ "use_peft": false,
10
+ "push_to_hub": false,
11
+ "repo_id": null,
12
+ "private": null,
13
+ "tags": null,
14
+ "license": null,
15
+ "pretrained_path": null,
16
+ "pretrained_revision": null,
17
+ "chunk_size": 32,
18
+ "n_action_steps": 32,
19
+ "max_action_dim": 64,
20
+ "normalization_mapping": {
21
+ "VISUAL": "IDENTITY",
22
+ "STATE": "IDENTITY",
23
+ "ACTION": "IDENTITY"
24
+ },
25
+ "mot_backbone": {
26
+ "transformers_version": "5.5.4",
27
+ "architectures": null,
28
+ "output_hidden_states": false,
29
+ "return_dict": true,
30
+ "dtype": null,
31
+ "chunk_size_feed_forward": 0,
32
+ "is_encoder_decoder": false,
33
+ "id2label": {
34
+ "0": "LABEL_0",
35
+ "1": "LABEL_1"
36
+ },
37
+ "label2id": {
38
+ "LABEL_0": 0,
39
+ "LABEL_1": 1
40
+ },
41
+ "problem_type": null,
42
+ "text_config": {
43
+ "architectures": null,
44
+ "output_hidden_states": false,
45
+ "return_dict": true,
46
+ "dtype": null,
47
+ "chunk_size_feed_forward": 0,
48
+ "is_encoder_decoder": false,
49
+ "id2label": {
50
+ "0": "LABEL_0",
51
+ "1": "LABEL_1"
52
+ },
53
+ "label2id": {
54
+ "LABEL_0": 0,
55
+ "LABEL_1": 1
56
+ },
57
+ "problem_type": null,
58
+ "vocab_size": 131072,
59
+ "hidden_size": 2048,
60
+ "intermediate_size": 9216,
61
+ "num_hidden_layers": 28,
62
+ "num_attention_heads": 16,
63
+ "num_key_value_heads": 8,
64
+ "head_dim": 128,
65
+ "hidden_act": "relu2",
66
+ "mlp_bias": false,
67
+ "attention_bias": false,
68
+ "attention_dropout": 0.0,
69
+ "max_position_embeddings": 131072,
70
+ "initializer_range": 0.02,
71
+ "rms_norm_eps": 1e-05,
72
+ "use_cache": true,
73
+ "rope_parameters": {
74
+ "mrope_section": [
75
+ 24,
76
+ 20,
77
+ 20
78
+ ],
79
+ "rope_theta": 100000000,
80
+ "rope_type": "default"
81
+ },
82
+ "pad_token_id": 0,
83
+ "joint_attn_implementation": "two_way",
84
+ "_name_or_path": "",
85
+ "eos_token_id": 11,
86
+ "model_type": "nemotron_3_dense_vl_text",
87
+ "output_attentions": false
88
+ },
89
+ "vision_config": {
90
+ "architectures": null,
91
+ "output_hidden_states": false,
92
+ "return_dict": true,
93
+ "dtype": null,
94
+ "chunk_size_feed_forward": 0,
95
+ "is_encoder_decoder": false,
96
+ "id2label": {
97
+ "0": "LABEL_0",
98
+ "1": "LABEL_1"
99
+ },
100
+ "label2id": {
101
+ "LABEL_0": 0,
102
+ "LABEL_1": 1
103
+ },
104
+ "problem_type": null,
105
+ "hidden_size": 1152,
106
+ "intermediate_size": 4304,
107
+ "num_hidden_layers": 27,
108
+ "num_attention_heads": 16,
109
+ "num_channels": 3,
110
+ "patch_size": 16,
111
+ "hidden_act": "gelu_pytorch_tanh",
112
+ "layer_norm_eps": 1e-06,
113
+ "attention_dropout": 0.0,
114
+ "num_patches": 256,
115
+ "spatial_merge_size": 2,
116
+ "_name_or_path": "",
117
+ "model_type": "nemotron_3_dense_vl_vision",
118
+ "output_attentions": false
119
+ },
120
+ "projector_config": {
121
+ "architectures": null,
122
+ "output_hidden_states": false,
123
+ "return_dict": true,
124
+ "dtype": null,
125
+ "chunk_size_feed_forward": 0,
126
+ "is_encoder_decoder": false,
127
+ "id2label": {
128
+ "0": "LABEL_0",
129
+ "1": "LABEL_1"
130
+ },
131
+ "label2id": {
132
+ "LABEL_0": 0,
133
+ "LABEL_1": 1
134
+ },
135
+ "problem_type": null,
136
+ "input_hidden_size": 1152,
137
+ "merger_intermediate_size": 11520,
138
+ "out_hidden_size": 2048,
139
+ "spatial_merge_size": 2,
140
+ "use_postshuffle_norm": false,
141
+ "_name_or_path": "",
142
+ "model_type": "nemotron_3_dense_vl_projector",
143
+ "output_attentions": false
144
+ },
145
+ "image_token_id": 19,
146
+ "video_token_id": 18,
147
+ "vision_start_token_id": 20,
148
+ "vision_end_token_id": 21,
149
+ "tie_word_embeddings": false,
150
+ "_name_or_path": "",
151
+ "model_type": "nemotron_3_dense_vl",
152
+ "output_attentions": false
153
+ },
154
+ "tokenizer_name": "nvidia/Cosmos3-Edge",
155
+ "vae": {
156
+ "base_dim": 160,
157
+ "decoder_base_dim": 256,
158
+ "z_dim": 48,
159
+ "dim_mult": [
160
+ 1,
161
+ 2,
162
+ 4,
163
+ 4
164
+ ],
165
+ "num_res_blocks": 2,
166
+ "attn_scales": [],
167
+ "temperal_downsample": [
168
+ false,
169
+ true,
170
+ true
171
+ ],
172
+ "dropout": 0.0,
173
+ "latents_mean": [
174
+ -0.2289,
175
+ -0.0052,
176
+ -0.1323,
177
+ -0.2339,
178
+ -0.2799,
179
+ 0.0174,
180
+ 0.1838,
181
+ 0.1557,
182
+ -0.1382,
183
+ 0.0542,
184
+ 0.2813,
185
+ 0.0891,
186
+ 0.157,
187
+ -0.0098,
188
+ 0.0375,
189
+ -0.1825,
190
+ -0.2246,
191
+ -0.1207,
192
+ -0.0698,
193
+ 0.5109,
194
+ 0.2665,
195
+ -0.2108,
196
+ -0.2158,
197
+ 0.2502,
198
+ -0.2055,
199
+ -0.0322,
200
+ 0.1109,
201
+ 0.1567,
202
+ -0.0729,
203
+ 0.0899,
204
+ -0.2799,
205
+ -0.123,
206
+ -0.0313,
207
+ -0.1649,
208
+ 0.0117,
209
+ 0.0723,
210
+ -0.2839,
211
+ -0.2083,
212
+ -0.052,
213
+ 0.3748,
214
+ 0.0152,
215
+ 0.1957,
216
+ 0.1433,
217
+ -0.2944,
218
+ 0.3573,
219
+ -0.0548,
220
+ -0.1681,
221
+ -0.0667
222
+ ],
223
+ "latents_std": [
224
+ 0.4765,
225
+ 1.0364,
226
+ 0.4514,
227
+ 1.1677,
228
+ 0.5313,
229
+ 0.499,
230
+ 0.4818,
231
+ 0.5013,
232
+ 0.8158,
233
+ 1.0344,
234
+ 0.5894,
235
+ 1.0901,
236
+ 0.6885,
237
+ 0.6165,
238
+ 0.8454,
239
+ 0.4978,
240
+ 0.5759,
241
+ 0.3523,
242
+ 0.7135,
243
+ 0.6804,
244
+ 0.5833,
245
+ 1.4146,
246
+ 0.8986,
247
+ 0.5659,
248
+ 0.7069,
249
+ 0.5338,
250
+ 0.4889,
251
+ 0.4917,
252
+ 0.4069,
253
+ 0.4999,
254
+ 0.6866,
255
+ 0.4093,
256
+ 0.5709,
257
+ 0.6065,
258
+ 0.6415,
259
+ 0.4944,
260
+ 0.5726,
261
+ 1.2042,
262
+ 0.5458,
263
+ 1.6887,
264
+ 0.3971,
265
+ 1.06,
266
+ 0.3943,
267
+ 0.5537,
268
+ 0.5444,
269
+ 0.4089,
270
+ 0.7468,
271
+ 0.7744
272
+ ],
273
+ "is_residual": true,
274
+ "in_channels": 12,
275
+ "out_channels": 12,
276
+ "patch_size": 2,
277
+ "scale_factor_temporal": 4,
278
+ "scale_factor_spatial": 16
279
+ },
280
+ "latent_patch_size": 2,
281
+ "timestep_scale": 0.001,
282
+ "timestep_embedding_dim": 256,
283
+ "num_embodiment_domains": 32,
284
+ "base_fps": 24.0,
285
+ "temporal_modality_margin": 15000,
286
+ "video_temporal_causal": false,
287
+ "train_gen_stream_only": true,
288
+ "freeze_vision_encoder": true,
289
+ "load_action_modules": true,
290
+ "gradient_checkpointing": false,
291
+ "compile_model": false,
292
+ "compile_mode": "default",
293
+ "attn_implementation": "sdpa",
294
+ "num_train_timesteps": 1000,
295
+ "video_time_distribution": "waver",
296
+ "training_shift": {
297
+ "256": 3.0,
298
+ "480": 5.0,
299
+ "720": 10.0
300
+ },
301
+ "independent_action_schedule": false,
302
+ "action_time_distribution": "logitnormal",
303
+ "action_training_shift": null,
304
+ "video_loss_weight": 10.0,
305
+ "action_loss_weight": 10.0,
306
+ "num_inference_steps": 4,
307
+ "guidance_scale": 3.0,
308
+ "guidance_interval": null,
309
+ "sampling_shift": 5.0,
310
+ "sampler": "unipc",
311
+ "save_predicted_video": false,
312
+ "mode": "wam",
313
+ "use_state": true,
314
+ "domain_id": 8,
315
+ "fps": null,
316
+ "view_size": null,
317
+ "view_description": "The top row is from the wrist-mounted camera. The bottom row contains two horizontally concatenated third-person perspective views of the scene from opposite sides, with the robot visible.",
318
+ "idle_state": true,
319
+ "action_feature_names": null,
320
+ "cfg_dropout_rate": 0.1,
321
+ "optimizer_lr": 8e-05,
322
+ "optimizer_betas": [
323
+ 0.9,
324
+ 0.99
325
+ ],
326
+ "optimizer_eps": 1e-08,
327
+ "optimizer_weight_decay": 0.05,
328
+ "optimizer_grad_clip_norm": 1.0,
329
+ "optimizer_action_lr_multiplier": 5.0,
330
+ "scheduler_warmup_steps": 0,
331
+ "scheduler_decay_steps": 100000
332
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9fdb549cd5e7f8418f40fc19d8d8e29b80d747e8a48f1b6e6cfc0f4838e03d2
3
+ size 9137003624
policy_postprocessor.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "policy_postprocessor",
3
+ "steps": [
4
+ {
5
+ "registry_name": "unnormalizer_processor",
6
+ "config": {
7
+ "eps": 1e-08,
8
+ "features": {},
9
+ "norm_map": {
10
+ "VISUAL": "IDENTITY",
11
+ "STATE": "IDENTITY",
12
+ "ACTION": "IDENTITY"
13
+ }
14
+ }
15
+ },
16
+ {
17
+ "registry_name": "device_processor",
18
+ "config": {
19
+ "device": "cuda",
20
+ "float_dtype": null
21
+ }
22
+ }
23
+ ]
24
+ }
policy_preprocessor.json ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "policy_preprocessor",
3
+ "steps": [
4
+ {
5
+ "registry_name": "rename_observations_processor",
6
+ "config": {
7
+ "rename_map": {}
8
+ }
9
+ },
10
+ {
11
+ "registry_name": "to_batch_processor",
12
+ "config": {}
13
+ },
14
+ {
15
+ "registry_name": "device_processor",
16
+ "config": {
17
+ "device": "cuda",
18
+ "float_dtype": null
19
+ }
20
+ },
21
+ {
22
+ "registry_name": "cosmos3_idle_frame_processor",
23
+ "config": {
24
+ "action_feature_names": null,
25
+ "fps": null,
26
+ "use_state": true,
27
+ "idle_state": true,
28
+ "translation_threshold": 0.005,
29
+ "rotation_threshold": 0.026179938779914945,
30
+ "gripper_threshold": 0.01,
31
+ "joint_threshold": 0.005,
32
+ "min_streak": 3
33
+ }
34
+ },
35
+ {
36
+ "registry_name": "normalizer_processor",
37
+ "config": {
38
+ "eps": 1e-08,
39
+ "features": {},
40
+ "norm_map": {
41
+ "VISUAL": "IDENTITY",
42
+ "STATE": "IDENTITY",
43
+ "ACTION": "IDENTITY"
44
+ }
45
+ }
46
+ },
47
+ {
48
+ "registry_name": "cosmos3_video_canvas_processor",
49
+ "config": {
50
+ "camera_keys": [],
51
+ "view_size": null,
52
+ "fps": null,
53
+ "chunk_size": 32,
54
+ "temporal_compression": 4,
55
+ "spatial_compression": 16
56
+ }
57
+ },
58
+ {
59
+ "registry_name": "cosmos3_prompt_processor",
60
+ "config": {
61
+ "view_description": "The top row is from the wrist-mounted camera. The bottom row contains two horizontally concatenated third-person perspective views of the scene from opposite sides, with the robot visible.",
62
+ "fps": null,
63
+ "chunk_size": 32,
64
+ "use_state": true
65
+ }
66
+ },
67
+ {
68
+ "registry_name": "cosmos3_tokenizer_processor",
69
+ "config": {
70
+ "tokenizer_name": "tokenizer",
71
+ "cfg_dropout_rate": 0.1
72
+ },
73
+ "artifacts": {
74
+ "tokenizer_name": "tokenizer"
75
+ }
76
+ },
77
+ {
78
+ "registry_name": "device_processor",
79
+ "config": {
80
+ "device": "cuda",
81
+ "float_dtype": null
82
+ }
83
+ }
84
+ ]
85
+ }
tokenizer/chat_template.jinja ADDED
@@ -0,0 +1,324 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ {% macro render_extra_keys(json_dict, handled_keys) %}
3
+ {%- if json_dict is mapping %}
4
+ {%- for json_key in json_dict if json_key not in handled_keys %}
5
+ {%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
6
+ {{- '
7
+ <' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
8
+ {%- else %}
9
+ {{-'
10
+ <' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
11
+ {%- endif %}
12
+ {%- endfor %}
13
+ {%- endif %}
14
+ {% endmacro %}
15
+ {%- set enable_thinking = enable_thinking if enable_thinking is defined else True %}
16
+ {%- set truncate_history_thinking = truncate_history_thinking if truncate_history_thinking is defined else True %}
17
+
18
+ {%- set ns = namespace(last_user_idx = -1) %}
19
+ {%- set loop_messages = messages %}
20
+ {%- for m in loop_messages %}
21
+ {%- if m["role"] == "user" %}
22
+ {%- set ns.last_user_idx = loop.index0 %}
23
+ {%- endif %}
24
+ {%- endfor %}
25
+
26
+ {%- if messages[0]["role"] == "system" %}
27
+ {%- if messages[0]["content"] is string %}
28
+ {%- set system_message = messages[0]["content"] %}
29
+ {%- else %}
30
+ {%- set ns = namespace(system_text="") %}
31
+ {%- for part in messages[0]["content"] %}
32
+ {%- if 'text' in part %}
33
+ {%- set ns.system_text = ns.system_text + part.text %}
34
+ {%- endif %}
35
+ {%- endfor %}
36
+ {%- set system_message = ns.system_text %}
37
+ {%- endif %}
38
+
39
+ {%- set loop_messages = messages[1:] %}
40
+ {%- else %}
41
+ {%- set system_message = "" %}
42
+ {%- set loop_messages = messages %}
43
+ {%- endif %}
44
+ {%- if not tools is defined %}
45
+ {%- set tools = [] %}
46
+ {%- endif %}
47
+ {# Recompute last_user_idx relative to loop_messages after handling system #}
48
+ {%- set ns = namespace(last_user_idx = -1) %}
49
+ {%- for m in loop_messages %}
50
+ {%- if m["role"] == "user" %}
51
+ {%- set ns.last_user_idx = loop.index0 %}
52
+ {%- endif %}
53
+ {%- endfor %}
54
+ {%- if system_message is defined %}
55
+ {{- "<|im_start|>system
56
+ " + system_message }}
57
+ {%- else %}
58
+ {%- if tools is iterable and tools | length > 0 %}
59
+ {{- "<|im_start|>system
60
+ " }}
61
+ {%- endif %}
62
+ {%- endif %}
63
+ {%- if tools is iterable and tools | length > 0 %}
64
+ {%- if system_message is defined and system_message | length > 0 %}
65
+ {{- "
66
+
67
+ " }}
68
+ {%- endif %}
69
+ {{- "# Tools
70
+
71
+ You have access to the following functions:
72
+
73
+ " }}
74
+ {{- "<tools>" }}
75
+ {%- for tool in tools %}
76
+ {%- if tool.function is defined %}
77
+ {%- set tool = tool.function %}
78
+ {%- endif %}
79
+ {{- "
80
+ <function>
81
+ <name>" ~ tool.name ~ "</name>" }}
82
+ {%- if tool.description is defined %}
83
+ {{- '
84
+ <description>' ~ (tool.description | trim) ~ '</description>' }}
85
+ {%- endif %}
86
+ {{- '
87
+ <parameters>' }}
88
+ {%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
89
+ {%- for param_name, param_fields in tool.parameters.properties|items %}
90
+ {{- '
91
+ <parameter>' }}
92
+ {{- '
93
+ <name>' ~ param_name ~ '</name>' }}
94
+ {%- if param_fields.type is defined %}
95
+ {{- '
96
+ <type>' ~ (param_fields.type | string) ~ '</type>' }}
97
+ {%- endif %}
98
+ {%- if param_fields.description is defined %}
99
+ {{- '
100
+ <description>' ~ (param_fields.description | trim) ~ '</description>' }}
101
+ {%- endif %}
102
+ {%- if param_fields.enum is defined %}
103
+ {{- '
104
+ <enum>' ~ (param_fields.enum | tojson | safe) ~ '</enum>' }}
105
+ {%- endif %}
106
+ {%- set handled_keys = ['name', 'type', 'description', 'enum'] %}
107
+ {{- render_extra_keys(param_fields, handled_keys) }}
108
+ {{- '
109
+ </parameter>' }}
110
+ {%- endfor %}
111
+ {%- endif %}
112
+ {% set handled_keys = ['type', 'properties', 'required'] %}
113
+ {{- render_extra_keys(tool.parameters, handled_keys) }}
114
+ {%- if tool.parameters is defined and tool.parameters.required is defined %}
115
+ {{- '
116
+ <required>' ~ (tool.parameters.required | tojson | safe) ~ '</required>' }}
117
+ {%- endif %}
118
+ {{- '
119
+ </parameters>' }}
120
+ {%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
121
+ {{- render_extra_keys(tool, handled_keys) }}
122
+ {{- '
123
+ </function>' }}
124
+ {%- endfor %}
125
+ {{- "
126
+ </tools>" }}
127
+
128
+ {{- '
129
+
130
+ If you choose to call a function ONLY reply in the following format with NO suffix:
131
+
132
+ <tool_call>
133
+ <function=example_function_name>
134
+ <parameter=example_parameter_1>
135
+ value_1
136
+ </parameter>
137
+ <parameter=example_parameter_2>
138
+ This is the value for the second parameter
139
+ that can span
140
+ multiple lines
141
+ </parameter>
142
+ </function>
143
+ </tool_call>
144
+
145
+ <IMPORTANT>
146
+ Reminder:
147
+ - Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags
148
+ - Required parameters MUST be specified
149
+ - You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after
150
+ - If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls
151
+ </IMPORTANT>' }}
152
+ {%- endif %}
153
+
154
+
155
+ {%- if system_message is defined %}
156
+ {{- '<|im_end|>
157
+ ' }}
158
+ {%- else %}
159
+ {%- if tools is iterable and tools | length > 0 %}
160
+ {{- '<|im_end|>
161
+ ' }}
162
+ {%- endif %}
163
+ {%- endif %}
164
+
165
+ {%- set image_count = namespace(value=0) %}
166
+ {%- set video_count = namespace(value=0) %}
167
+ {%- for message in loop_messages %}
168
+ {%- if message.role == "assistant" %}
169
+ {# Add reasoning content in to content field for unified processing below. #}
170
+ {%- set text_content = namespace(text_content="") -%}
171
+ {%- if message.content is string %}
172
+ {%- set text_content.text_content = message.content %}
173
+ {%- else %}
174
+ {%- for content in message.content %}
175
+ {%- if 'text' in content %}
176
+ {%- set text_content.text_content = content.text %}
177
+ {%- break %}
178
+ {%- endif %}
179
+ {%- endfor %}
180
+ {%- endif %}
181
+ {%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %}
182
+ {%- set content = "<think>
183
+ " ~ message.reasoning_content ~ "
184
+ </think>
185
+ " ~ (text_content.text_content | default('', true)) %}
186
+ {%- else %}
187
+ {%- set content = text_content.text_content | default('', true) %}
188
+ {# Allow downstream logic to to take care of broken thought, only handle coherent reasoning here. #}
189
+ {%- if '<think>' not in content and '</think>' not in content %}
190
+ {%- set content = "<think></think>" ~ content %}
191
+ {%- else %}
192
+ {%- set content = content %}
193
+ {%- endif -%}
194
+ {%- endif %}
195
+ {%- if message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
196
+ {# Assistant message has tool calls. #}
197
+ {{- '<|im_start|>assistant
198
+ ' }}
199
+ {%- set include_content = not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
200
+ {%- if content is string and content | trim | length > 0 %}
201
+ {%- if include_content %}
202
+ {{- (content | trim) ~ '
203
+ ' -}}
204
+ {%- else %}
205
+ {%- set c = (content | string) %}
206
+ {%- if '</think>' in c %}
207
+ {# Keep only content after the last closing think. Also generation prompt causes this. #}
208
+ {%- set c = c.split('</think>')[-1] %}
209
+ {%- elif '<think>' in c %}
210
+ {# If <think> was opened but never closed, drop the trailing think segment #}
211
+ {%- set c = c.split('<think>')[0] %}
212
+ {%- endif %}
213
+ {%- set c = "<think></think>" ~ c | trim %}
214
+ {%- if c | length > 0 %}
215
+ {{- c ~ '
216
+ ' -}}
217
+ {%- endif %}
218
+ {%- endif %}
219
+ {%- else %}
220
+ {{- "<think></think>" -}}
221
+ {%- endif %}
222
+ {%- for tool_call in message.tool_calls %}
223
+ {%- if tool_call.function is defined %}
224
+ {%- set tool_call = tool_call.function %}
225
+ {%- endif %}
226
+ {{- '<tool_call>
227
+ <function=' ~ tool_call.name ~ '>
228
+ ' -}}
229
+ {%- if tool_call.arguments is defined %}
230
+ {%- for args_name, args_value in tool_call.arguments|items %}
231
+ {{- '<parameter=' ~ args_name ~ '>
232
+ ' -}}
233
+ {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
234
+ {{- args_value ~ '
235
+ </parameter>
236
+ ' -}}
237
+ {%- endfor %}
238
+ {%- endif %}
239
+ {{- '</function>
240
+ </tool_call>
241
+ ' -}}
242
+ {%- endfor %}
243
+ {{- '<|im_end|>
244
+ ' }}
245
+ {%- else %}
246
+ {# Assistant message doesn't have tool calls. #}
247
+ {%- if not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
248
+ {{- '<|im_start|>assistant
249
+ ' ~ (content | default('', true) | string | trim) ~ '<|im_end|>
250
+ ' }}
251
+ {%- else %}
252
+ {%- set c = (content | default('', true) | string) %}
253
+ {%- if '<think>' in c and '</think>' in c %}
254
+ {%- set c = "<think></think>" ~ c.split('</think>')[-1] %}
255
+ {%- endif %}
256
+ {%- set c = c | trim %}
257
+ {%- if c | length > 0 %}
258
+ {{- '<|im_start|>assistant
259
+ ' ~ c ~ '<|im_end|>
260
+ ' }}
261
+ {%- else %}
262
+ {{- '<|im_start|>assistant
263
+ <|im_end|>
264
+ ' }}
265
+ {%- endif %}
266
+ {%- endif %}
267
+ {%- endif %}
268
+ {%- elif message.role == "user" or message.role == "system" %}
269
+ {{- '<|im_start|>' + message.role + '
270
+ ' }}
271
+ {%- if message.content is string %}
272
+ {{- message.content }}
273
+ {%- else %}
274
+ {%- for content in message.content %}
275
+ {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
276
+ {%- set image_count.value = image_count.value + 1 %}
277
+ {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
278
+ <|vision_start|><|image_pad|><|vision_end|>
279
+ {%- elif content.type == 'video' or 'video' in content %}
280
+ {%- set video_count.value = video_count.value + 1 %}
281
+ {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
282
+ <|vision_start|><|video_pad|><|vision_end|>
283
+ {%- elif 'text' in content %}
284
+ {{- content.text }}
285
+ {%- endif %}
286
+ {%- endfor %}
287
+ {%- endif %}
288
+ {{- '<|im_end|>
289
+ ' }}
290
+ {%- elif message.role == "tool" %}
291
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
292
+ {{- '<|im_start|>user
293
+ ' }}
294
+ {%- endif %}
295
+ {{- '<tool_response>
296
+ ' }}
297
+ {{- message.content }}
298
+ {{- '
299
+ </tool_response>
300
+ ' }}
301
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
302
+ {{- '<|im_end|>
303
+ ' }}
304
+ {%- elif loop.last %}
305
+ {{- '<|im_end|>
306
+ ' }}
307
+ {%- endif %}
308
+ {%- else %}
309
+ {{- '<|im_start|>' + message.role + '
310
+ ' + message.content + '<|im_end|>
311
+ ' }}
312
+ {%- endif %}
313
+ {%- endfor %}
314
+
315
+ {%- if add_generation_prompt %}
316
+ {%- if enable_thinking %}
317
+ {{- '<|im_start|>assistant
318
+ <think>
319
+ ' }}
320
+ {%- else %}
321
+ {{- '<|im_start|>assistant
322
+ <think></think>' }}
323
+ {%- endif %}
324
+ {%- endif %}
tokenizer/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4dc692a99dca6d92a44e9529ffb53541eb79bb260eec7281bc51268c05d7311c
3
+ size 17077500
tokenizer/tokenizer_config.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<s>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "is_local": true,
8
+ "model_input_names": [
9
+ "input_ids",
10
+ "attention_mask"
11
+ ],
12
+ "model_max_length": 262144,
13
+ "pad_token": "<|im_end|>",
14
+ "return_mm_token_type_ids": true,
15
+ "tokenizer_class": "TokenizersBackend",
16
+ "unk_token": "<unk>"
17
+ }