| --- |
| license: other |
| license_name: nvidia-open-model-license |
| license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ |
| tags: |
| - robotics |
| - franka |
| - manipulation |
| - video-generation |
| - inverse-dynamics |
| - lora |
| --- |
| |
| # NT_checkpoint |
| |
| Two models for a Franka cube-stacking task. A video model generates multi-view footage of |
| the arm stacking three cubes; an inverse dynamics model reads video back and recovers the |
| actions that would produce it. |
| |
| | folder | model | size | base | |
| |---|---|---|---| |
| | [`vgm_lora/`](vgm_lora) | rank-32 LoRA, video generation | 288 MB | `nvidia/Cosmos-Predict2-14B-Sample-GR00T-Dreams-DROID` | |
| | [`idm/`](idm) | inverse dynamics, full fine-tune | 2.4 GB | `seonghyeonye/IDM_franka` | |
|
|
| Each folder has its own README with the details. Start there. |
|
|
| ## Quick orientation |
|
|
| **`vgm_lora/`** — adapter weights only. Attach to the base Cosmos-Predict2 checkpoint with |
| rank 32, alpha 32, targets `q_proj,k_proj,v_proj,output_proj,mlp.layer1,mlp.layer2`. |
| Conditioned on one 768×432 four-view frame plus a prompt, it produces a 93-frame clip at |
| 16 fps. |
| |
| **`idm/`** — load with `IDM.from_pretrained()`. One forward pass sees two frames, `s` and |
| `s+16`, from three cameras and emits 16 actions. **Its gripper output needs |
| post-processing** — see [`idm/README.md`](idm/README.md#required-post-processing). Reading |
| `action[12]` raw gives a vote tally, not a gripper state. |
|
|
| ## Scope |
|
|
| One task, one scene: three 5 cm cubes on a white table, a fixed three-camera rig, a Franka |
| arm. Both models are specific to that setup. |
|
|
| ## Licence |
|
|
| Both are **derivatives of NVIDIA-licensed weights** and the upstream terms govern. |
| Attribution to NVIDIA Cosmos-Predict2 and to the GR00T-Dreams IDM is required. The |
| `cosmos-predict2` repository code is Apache-2.0, which is separate from the weights licence. |
|
|
