| GL backend: egl; seeds: checked against 926 dataset seeds: no overlap (max training seed 1200300) |
| /root/work/Isaac-GR00T/.venv/lib/python3.12/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. |
| check_for_updates() |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| /root/work/Isaac-GR00T/gr00t/model/modules/dit.py:278: FutureWarning: Accessing config attribute `compute_dtype` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'compute_dtype' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.compute_dtype'. |
| embedding_dim=self.inner_dim, compute_dtype=self.compute_dtype |
| /root/work/Isaac-GR00T/gr00t/model/modules/dit.py:309: FutureWarning: Accessing config attribute `output_dim` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'output_dim' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.output_dim'. |
| self.proj_out_2 = nn.Linear(self.inner_dim, self.output_dim) |
| Total number of DiT parameters: 1091722240 |
| Total number of SelfAttentionTransformer parameters: 201433088 |
|
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 44.04it/s] |
| [push 1/50] OK 'push the yellow block onto the green pad' steps=78 |
| [push 2/50] OK 'push the yellow block onto the blue pad' steps=104 |
| [push 3/50] OK 'push the orange block onto the red pad' steps=117 |
| [push 4/50] OK 'push the green block onto the blue pad' steps=88 |
| [push 5/50] OK 'push the orange block onto the yellow pad' steps=85 |
| [push 6/50] OK 'push the blue block onto the orange pad' steps=55 |
| [push 7/50] OK 'push the blue block onto the blue pad' steps=192 |
| [push 8/50] OK 'push the yellow block onto the blue pad' steps=160 |
| [push 9/50] OK 'push the purple block onto the red pad' steps=75 |
| [push 10/50] OK 'push the purple block onto the yellow pad' steps=88 |
| [push 11/50] OK 'push the blue block onto the purple pad' steps=76 |
| [push 12/50] FAIL 'push the red block onto the yellow pad' steps=440 target_short_of_pad |
| [push 13/50] FAIL 'push the green block onto the purple pad' steps=440 target_short_of_pad |
| [push 14/50] OK 'push the yellow block onto the yellow pad' steps=75 |
| [push 15/50] OK 'push the red block onto the yellow pad' steps=54 |
| [push 16/50] FAIL 'push the red block onto the yellow pad' steps=440 target_not_moved |
| [push 17/50] OK 'push the blue block onto the red pad' steps=73 |
| [push 18/50] FAIL 'push the red block onto the purple pad' steps=440 target_short_of_pad |
| [push 19/50] OK 'push the red block onto the orange pad' steps=82 |
| [push 20/50] OK 'push the orange block onto the blue pad' steps=84 |
| [push 21/50] OK 'push the red block onto the red pad' steps=76 |
| [push 22/50] OK 'push the green block onto the red pad' steps=88 |
| [push 23/50] OK 'push the purple block onto the orange pad' steps=74 |
| [push 24/50] OK 'push the orange block onto the red pad' steps=72 |
| [push 25/50] OK 'push the orange block onto the blue pad' steps=60 |
| [push 26/50] OK 'push the blue block onto the green pad' steps=55 |
| [push 27/50] FAIL 'push the purple block onto the orange pad' steps=440 target_short_of_pad |
| [push 28/50] FAIL 'push the purple block onto the orange pad' steps=440 target_not_moved |
| [push 29/50] OK 'push the orange block onto the yellow pad' steps=174 |
| [push 30/50] FAIL 'push the red block onto the red pad' steps=440 target_not_moved |
| [push 31/50] OK 'push the yellow block onto the red pad' steps=87 |
| [push 32/50] OK 'push the orange block onto the purple pad' steps=85 |
| [push 33/50] OK 'push the purple block onto the blue pad' steps=63 |
| [push 34/50] OK 'push the purple block onto the orange pad' steps=96 |
| [push 35/50] OK 'push the blue block onto the green pad' steps=74 |
| [push 36/50] OK 'push the green block onto the green pad' steps=78 |
| [push 37/50] OK 'push the yellow block onto the blue pad' steps=77 |
| [push 38/50] OK 'push the yellow block onto the green pad' steps=92 |
| [push 39/50] OK 'push the green block onto the green pad' steps=76 |
| [push 40/50] FAIL 'push the yellow block onto the green pad' steps=440 target_short_of_pad |
| [push 41/50] OK 'push the purple block onto the orange pad' steps=92 |
| [push 42/50] OK 'push the red block onto the purple pad' steps=95 |
| [push 43/50] OK 'push the purple block onto the green pad' steps=72 |
| [push 44/50] OK 'push the purple block onto the purple pad' steps=45 |
| [push 45/50] OK 'push the yellow block onto the purple pad' steps=97 |
| [push 46/50] OK 'push the red block onto the yellow pad' steps=65 |
| [push 47/50] OK 'push the purple block onto the red pad' steps=75 |
| [push 48/50] OK 'push the blue block onto the orange pad' steps=58 |
| [push 49/50] OK 'push the blue block onto the yellow pad' steps=110 |
| [push 50/50] OK 'push the blue block onto the blue pad' steps=156 |
| [button 1/50] OK 'press the blue button' steps=68 |
| [button 2/50] OK 'press the orange button' steps=48 |
| [button 3/50] OK 'press the purple button' steps=53 |
| [button 4/50] FAIL 'press the orange button' steps=260 no_press |
| [button 5/50] FAIL 'press the blue button' steps=260 no_press |
| [button 6/50] FAIL 'press the purple button' steps=260 no_press |
| [button 7/50] OK 'press the red button' steps=112 |
| [button 8/50] OK 'press the blue button' steps=49 |
| [button 9/50] OK 'press the red button' steps=63 |
| [button 10/50] OK 'press the yellow button' steps=47 |
| [button 11/50] FAIL 'press the red button' steps=260 no_press |
| [button 12/50] OK 'press the purple button' steps=75 |
| [button 13/50] OK 'press the red button' steps=246 |
| [button 14/50] FAIL 'press the purple button' steps=260 wrong_button_pressed |
| [button 15/50] FAIL 'press the green button' steps=260 partial_press |
| [button 16/50] OK 'press the red button' steps=57 |
| [button 17/50] FAIL 'press the blue button' steps=260 partial_press |
| [button 18/50] OK 'press the purple button' steps=63 |
| [button 19/50] FAIL 'press the yellow button' steps=260 wrong_button_pressed |
| [button 20/50] OK 'press the red button' steps=53 |
| [button 21/50] OK 'press the red button' steps=52 |
| [button 22/50] OK 'press the purple button' steps=94 |
| [button 23/50] FAIL 'press the yellow button' steps=260 partial_press |
| [button 24/50] OK 'press the green button' steps=53 |
| [button 25/50] FAIL 'press the blue button' steps=260 no_press |
| [button 26/50] OK 'press the yellow button' steps=53 |
| [button 27/50] OK 'press the yellow button' steps=48 |
| [button 28/50] OK 'press the green button' steps=49 |
| [button 29/50] OK 'press the yellow button' steps=138 |
| [button 30/50] OK 'press the blue button' steps=49 |
| [button 31/50] OK 'press the red button' steps=44 |
| [button 32/50] FAIL 'press the red button' steps=260 partial_press |
| [button 33/50] OK 'press the yellow button' steps=56 |
| [button 34/50] FAIL 'press the purple button' steps=260 no_press |
| [button 35/50] OK 'press the orange button' steps=82 |
| [button 36/50] OK 'press the red button' steps=68 |
| [button 37/50] FAIL 'press the red button' steps=260 partial_press |
| [button 38/50] OK 'press the purple button' steps=70 |
| [button 39/50] OK 'press the purple button' steps=50 |
| [button 40/50] FAIL 'press the red button' steps=260 no_press |
| [button 41/50] OK 'press the yellow button' steps=53 |
| [button 42/50] OK 'press the orange button' steps=41 |
| [button 43/50] OK 'press the blue button' steps=58 |
| [button 44/50] FAIL 'press the yellow button' steps=260 no_press |
| [button 45/50] OK 'press the red button' steps=57 |
| [button 46/50] FAIL 'press the orange button' steps=260 no_press |
| [button 47/50] OK 'press the purple button' steps=40 |
| [button 48/50] FAIL 'press the green button' steps=260 no_press |
| [button 49/50] FAIL 'press the blue button' steps=260 partial_press |
| [button 50/50] OK 'press the purple button' steps=40 |
| [box 1/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 2/50] FAIL 'lift the box and put it on the shelf' steps=440 tipped |
| [box 3/50] OK 'lift the box and put it on the shelf' steps=278 |
| [box 4/50] OK 'lift the box and put it on the shelf' steps=228 |
| [box 5/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 6/50] FAIL 'lift the box and put it on the shelf' steps=440 off_shelf_footprint_or_moving |
| [box 7/50] OK 'lift the box and put it on the shelf' steps=297 |
| [box 8/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 9/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_off_table |
| [box 10/50] OK 'lift the box and put it on the shelf' steps=229 |
| [box 11/50] OK 'lift the box and put it on the shelf' steps=236 |
| [box 12/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 13/50] OK 'lift the box and put it on the shelf' steps=339 |
| [box 14/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 15/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 16/50] OK 'lift the box and put it on the shelf' steps=440 |
| [box 17/50] OK 'lift the box and put it on the shelf' steps=244 |
| [box 18/50] OK 'lift the box and put it on the shelf' steps=243 |
| [box 19/50] OK 'lift the box and put it on the shelf' steps=186 |
| [box 20/50] OK 'lift the box and put it on the shelf' steps=190 |
| [box 21/50] OK 'lift the box and put it on the shelf' steps=220 |
| [box 22/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 23/50] OK 'lift the box and put it on the shelf' steps=207 |
| [box 24/50] OK 'lift the box and put it on the shelf' steps=258 |
| [box 25/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 26/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 27/50] OK 'lift the box and put it on the shelf' steps=326 |
| [box 28/50] OK 'lift the box and put it on the shelf' steps=197 |
| [box 29/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 30/50] OK 'lift the box and put it on the shelf' steps=365 |
| [box 31/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 32/50] FAIL 'lift the box and put it on the shelf' steps=440 off_shelf_footprint_or_moving |
| [box 33/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 34/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 35/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 36/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 37/50] OK 'lift the box and put it on the shelf' steps=277 |
| [box 38/50] FAIL 'lift the box and put it on the shelf' steps=440 not_released |
| [box 39/50] OK 'lift the box and put it on the shelf' steps=314 |
| [box 40/50] OK 'lift the box and put it on the shelf' steps=238 |
| [box 41/50] OK 'lift the box and put it on the shelf' steps=414 |
| [box 42/50] OK 'lift the box and put it on the shelf' steps=217 |
| [box 43/50] OK 'lift the box and put it on the shelf' steps=314 |
| [box 44/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 45/50] OK 'lift the box and put it on the shelf' steps=195 |
| [box 46/50] OK 'lift the box and put it on the shelf' steps=325 |
| [box 47/50] OK 'lift the box and put it on the shelf' steps=254 |
| [box 48/50] FAIL 'lift the box and put it on the shelf' steps=440 tipped |
| [box 49/50] OK 'lift the box and put it on the shelf' steps=230 |
| [box 50/50] OK 'lift the box and put it on the shelf' steps=405 |
| # Closed-loop evaluation: groot (/root/work/runs/asimov_n17/checkpoint-10000) |
|
|
| 50 held-out episodes per task (seeds 2**40 + task_id * 2**20 + k; checked against 926 dataset seeds: no overlap (max training seed 1200300)), 8 of 16 actions executed per inference at 20 Hz, success held 10 steps, GL egl. |
|
|
| | Task | Success | 95% CI | Mean time to success | Failures | |
| | |
| | push | 42/50 (84%) | 0.71-0.92 | 3.93 s | target_short_of_pad 5, target_not_moved 3 | |
| | button | 32/50 (64%) | 0.50-0.76 | 2.88 s | no_press 10, partial_press 6, wrong_button_pressed 2 | |
| | box | 28/50 (56%) | 0.42-0.69 | 13.26 s | dropped_or_not_placed 16, tipped 2, off_shelf_footprint_or_moving 2, dropped_off_table 1, not_released 1 | |
|
|
| Policy calls: {'mean': 0.095, 'p50': 0.0943, 'p90': 0.096, 'calls': 3970} |
|
|
| ## push: per instruction |
|
|
| | Instruction | Success | |
| | |
| | push the blue block onto the blue pad | 2/2 | |
| | push the blue block onto the green pad | 2/2 | |
| | push the blue block onto the orange pad | 2/2 | |
| | push the blue block onto the purple pad | 1/1 | |
| | push the blue block onto the red pad | 1/1 | |
| | push the blue block onto the yellow pad | 1/1 | |
| | push the green block onto the blue pad | 1/1 | |
| | push the green block onto the green pad | 2/2 | |
| | push the green block onto the purple pad | 0/1 | |
| | push the green block onto the red pad | 1/1 | |
| | push the orange block onto the blue pad | 2/2 | |
| | push the orange block onto the purple pad | 1/1 | |
| | push the orange block onto the red pad | 2/2 | |
| | push the orange block onto the yellow pad | 2/2 | |
| | push the purple block onto the blue pad | 1/1 | |
| | push the purple block onto the green pad | 1/1 | |
| | push the purple block onto the orange pad | 3/5 | |
| | push the purple block onto the purple pad | 1/1 | |
| | push the purple block onto the red pad | 2/2 | |
| | push the purple block onto the yellow pad | 1/1 | |
| | push the red block onto the orange pad | 1/1 | |
| | push the red block onto the purple pad | 1/2 | |
| | push the red block onto the red pad | 1/2 | |
| | push the red block onto the yellow pad | 2/4 | |
| | push the yellow block onto the blue pad | 3/3 | |
| | push the yellow block onto the green pad | 2/3 | |
| | push the yellow block onto the purple pad | 1/1 | |
| | push the yellow block onto the red pad | 1/1 | |
| | push the yellow block onto the yellow pad | 1/1 | |
|
|
| Videos: videos/push_success_seed1099511627776.mp4, videos/push_failure_seed1099511627787.mp4 |
|
|
| ## button: per instruction |
|
|
| | Instruction | Success | |
| | |
| | press the blue button | 4/8 | |
| | press the green button | 2/4 | |
| | press the orange button | 3/5 | |
| | press the purple button | 8/11 | |
| | press the red button | 9/13 | |
| | press the yellow button | 6/9 | |
|
|
| Videos: videos/button_success_seed1099512676352.mp4, videos/button_failure_seed1099512676355.mp4 |
|
|
| ## box: per instruction |
|
|
| | Instruction | Success | |
| | |
| | lift the box and put it on the shelf | 28/50 | |
|
|
| Videos: videos/box_failure_seed1099513724928.mp4, videos/box_success_seed1099513724930.mp4 |
|
|
|
|
