| GL backend: egl; seeds: checked against 926 dataset seeds: no overlap (max training seed 1200300) |
| /root/work/Isaac-GR00T/.venv/lib/python3.12/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. |
| check_for_updates() |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| /root/work/Isaac-GR00T/gr00t/model/modules/dit.py:278: FutureWarning: Accessing config attribute `compute_dtype` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'compute_dtype' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.compute_dtype'. |
| embedding_dim=self.inner_dim, compute_dtype=self.compute_dtype |
| /root/work/Isaac-GR00T/gr00t/model/modules/dit.py:309: FutureWarning: Accessing config attribute `output_dim` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'output_dim' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.output_dim'. |
| self.proj_out_2 = nn.Linear(self.inner_dim, self.output_dim) |
| Total number of DiT parameters: 1091722240 |
| Total number of SelfAttentionTransformer parameters: 201433088 |
|
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 24.03it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 23.99it/s] |
| [push 1/50] FAIL 'push the yellow block onto the green pad' steps=440 target_not_moved |
| [push 2/50] OK 'push the yellow block onto the blue pad' steps=136 |
| [push 3/50] FAIL 'push the orange block onto the red pad' steps=440 target_not_moved |
| [push 4/50] OK 'push the green block onto the blue pad' steps=49 |
| [push 5/50] OK 'push the orange block onto the yellow pad' steps=418 |
| [push 6/50] OK 'push the blue block onto the orange pad' steps=273 |
| [push 7/50] OK 'push the blue block onto the blue pad' steps=56 |
| [push 8/50] OK 'push the yellow block onto the blue pad' steps=106 |
| [push 9/50] FAIL 'push the purple block onto the red pad' steps=440 target_short_of_pad |
| [push 10/50] FAIL 'push the purple block onto the yellow pad' steps=440 target_short_of_pad |
| [push 11/50] FAIL 'push the blue block onto the purple pad' steps=440 target_not_moved |
| [push 12/50] FAIL 'push the red block onto the yellow pad' steps=440 target_on_wrong_pad |
| [push 13/50] FAIL 'push the green block onto the purple pad' steps=440 target_short_of_pad |
| [push 14/50] FAIL 'push the yellow block onto the yellow pad' steps=440 target_not_moved |
| [push 15/50] OK 'push the red block onto the yellow pad' steps=66 |
| [push 16/50] FAIL 'push the red block onto the yellow pad' steps=440 target_not_moved |
| [push 17/50] OK 'push the blue block onto the red pad' steps=127 |
| [push 18/50] FAIL 'push the red block onto the purple pad' steps=440 target_short_of_pad |
| [push 19/50] OK 'push the red block onto the orange pad' steps=71 |
| [push 20/50] OK 'push the orange block onto the blue pad' steps=134 |
| [push 21/50] FAIL 'push the red block onto the red pad' steps=440 target_not_moved |
| [push 22/50] FAIL 'push the green block onto the red pad' steps=440 target_not_moved |
| [push 23/50] OK 'push the purple block onto the orange pad' steps=132 |
| [push 24/50] OK 'push the orange block onto the red pad' steps=131 |
| [push 25/50] FAIL 'push the orange block onto the blue pad' steps=440 target_not_moved |
| [push 26/50] OK 'push the blue block onto the green pad' steps=77 |
| [push 27/50] OK 'push the purple block onto the orange pad' steps=101 |
| [push 28/50] OK 'push the purple block onto the orange pad' steps=46 |
| [push 29/50] OK 'push the orange block onto the yellow pad' steps=269 |
| [push 30/50] FAIL 'push the red block onto the red pad' steps=440 target_not_moved |
| [push 31/50] FAIL 'push the yellow block onto the red pad' steps=440 target_not_moved |
| [push 32/50] FAIL 'push the orange block onto the purple pad' steps=440 target_short_of_pad |
| [push 33/50] OK 'push the purple block onto the blue pad' steps=137 |
| [push 34/50] OK 'push the purple block onto the orange pad' steps=114 |
| [push 35/50] FAIL 'push the blue block onto the green pad' steps=440 target_not_moved |
| [push 36/50] FAIL 'push the green block onto the green pad' steps=440 target_not_moved |
| [push 37/50] FAIL 'push the yellow block onto the blue pad' steps=440 target_short_of_pad |
| [push 38/50] FAIL 'push the yellow block onto the green pad' steps=440 target_not_moved |
| [push 39/50] OK 'push the green block onto the green pad' steps=122 |
| [push 40/50] OK 'push the yellow block onto the green pad' steps=104 |
| [push 41/50] FAIL 'push the purple block onto the orange pad' steps=440 target_short_of_pad |
| [push 42/50] FAIL 'push the red block onto the purple pad' steps=440 target_short_of_pad |
| [push 43/50] OK 'push the purple block onto the green pad' steps=61 |
| [push 44/50] FAIL 'push the purple block onto the purple pad' steps=440 target_not_moved |
| [push 45/50] FAIL 'push the yellow block onto the purple pad' steps=440 target_not_moved |
| [push 46/50] OK 'push the red block onto the yellow pad' steps=80 |
| [push 47/50] FAIL 'push the purple block onto the red pad' steps=440 target_not_moved |
| [push 48/50] OK 'push the blue block onto the orange pad' steps=78 |
| [push 49/50] FAIL 'push the blue block onto the yellow pad' steps=440 target_short_of_pad |
| [push 50/50] FAIL 'push the blue block onto the blue pad' steps=440 target_not_moved |
| [button 1/50] FAIL 'press the blue button' steps=260 no_press |
| [button 2/50] FAIL 'press the orange button' steps=260 no_press |
| [button 3/50] FAIL 'press the purple button' steps=260 no_press |
| [button 4/50] OK 'press the orange button' steps=64 |
| [button 5/50] FAIL 'press the blue button' steps=260 wrong_button_pressed |
| [button 6/50] FAIL 'press the purple button' steps=260 no_press |
| [button 7/50] OK 'press the red button' steps=77 |
| [button 8/50] FAIL 'press the blue button' steps=260 no_press |
| [button 9/50] FAIL 'press the red button' steps=260 wrong_button_pressed |
| [button 10/50] FAIL 'press the yellow button' steps=260 partial_press |
| [button 11/50] FAIL 'press the red button' steps=260 no_press |
| [button 12/50] OK 'press the purple button' steps=64 |
| [button 13/50] FAIL 'press the red button' steps=260 no_press |
| [button 14/50] OK 'press the purple button' steps=68 |
| [button 15/50] OK 'press the green button' steps=43 |
| [button 16/50] FAIL 'press the red button' steps=260 no_press |
| [button 17/50] OK 'press the blue button' steps=42 |
| [button 18/50] FAIL 'press the purple button' steps=260 no_press |
| [button 19/50] FAIL 'press the yellow button' steps=260 wrong_button_pressed |
| [button 20/50] FAIL 'press the red button' steps=260 no_press |
| [button 21/50] OK 'press the red button' steps=45 |
| [button 22/50] FAIL 'press the purple button' steps=260 no_press |
| [button 23/50] FAIL 'press the yellow button' steps=260 no_press |
| [button 24/50] FAIL 'press the green button' steps=260 no_press |
| [button 25/50] OK 'press the blue button' steps=45 |
| [button 26/50] FAIL 'press the yellow button' steps=260 no_press |
| [button 27/50] OK 'press the yellow button' steps=68 |
| [button 28/50] OK 'press the green button' steps=42 |
| [button 29/50] FAIL 'press the yellow button' steps=260 no_press |
| [button 30/50] OK 'press the blue button' steps=101 |
| [button 31/50] FAIL 'press the red button' steps=260 no_press |
| [button 32/50] FAIL 'press the red button' steps=260 no_press |
| [button 33/50] OK 'press the yellow button' steps=43 |
| [button 34/50] FAIL 'press the purple button' steps=260 no_press |
| [button 35/50] OK 'press the orange button' steps=67 |
| [button 36/50] OK 'press the red button' steps=51 |
| [button 37/50] OK 'press the red button' steps=55 |
| [button 38/50] OK 'press the purple button' steps=40 |
| [button 39/50] FAIL 'press the purple button' steps=260 no_press |
| [button 40/50] FAIL 'press the red button' steps=260 no_press |
| [button 41/50] FAIL 'press the yellow button' steps=260 no_press |
| [button 42/50] FAIL 'press the orange button' steps=260 no_press |
| [button 43/50] FAIL 'press the blue button' steps=260 no_press |
| [button 44/50] FAIL 'press the yellow button' steps=260 no_press |
| [button 45/50] FAIL 'press the red button' steps=260 no_press |
| [button 46/50] OK 'press the orange button' steps=44 |
| [button 47/50] FAIL 'press the purple button' steps=260 no_press |
| [button 48/50] OK 'press the green button' steps=37 |
| [button 49/50] FAIL 'press the blue button' steps=260 no_press |
| [button 50/50] FAIL 'press the purple button' steps=260 no_press |
| [box 1/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 2/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 3/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 4/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 5/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 6/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 7/50] OK 'lift the box and put it on the shelf' steps=180 |
| [box 8/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 9/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 10/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 11/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 12/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 13/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 14/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 15/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 16/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 17/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 18/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 19/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 20/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 21/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 22/50] FAIL 'lift the box and put it on the shelf' steps=440 tipped |
| [box 23/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 24/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 25/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 26/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 27/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 28/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 29/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 30/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 31/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 32/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 33/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 34/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 35/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 36/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 37/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 38/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 39/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 40/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 41/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 42/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 43/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 44/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 45/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 46/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 47/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 48/50] FAIL 'lift the box and put it on the shelf' steps=440 not_lifted |
| [box 49/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| [box 50/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed |
| # Closed-loop evaluation: groot (/root/work/runs/asimov_n17/checkpoint-6000) |
|
|
| 50 held-out episodes per task (seeds 2**40 + task_id * 2**20 + k; checked against 926 dataset seeds: no overlap (max training seed 1200300)), 8 of 16 actions executed per inference at 20 Hz, success held 10 steps, GL egl. |
|
|
| | Task | Success | 95% CI | Mean time to success | Failures | |
| | |
| | push | 23/50 (46%) | 0.33-0.60 | 5.83 s | target_not_moved 17, target_short_of_pad 9, target_on_wrong_pad 1 | |
| | button | 18/50 (36%) | 0.24-0.50 | 2.32 s | no_press 28, wrong_button_pressed 3, partial_press 1 | |
| | box | 1/50 (2%) | 0.00-0.10 | 8.55 s | not_lifted 30, dropped_or_not_placed 18, tipped 1 | |
|
|
| Policy calls: {'mean': 0.0944, 'p50': 0.0941, 'p90': 0.0954, 'calls': 5762} |
|
|
| ## push: per instruction |
|
|
| | Instruction | Success | |
| | |
| | push the blue block onto the blue pad | 1/2 | |
| | push the blue block onto the green pad | 1/2 | |
| | push the blue block onto the orange pad | 2/2 | |
| | push the blue block onto the purple pad | 0/1 | |
| | push the blue block onto the red pad | 1/1 | |
| | push the blue block onto the yellow pad | 0/1 | |
| | push the green block onto the blue pad | 1/1 | |
| | push the green block onto the green pad | 1/2 | |
| | push the green block onto the purple pad | 0/1 | |
| | push the green block onto the red pad | 0/1 | |
| | push the orange block onto the blue pad | 1/2 | |
| | push the orange block onto the purple pad | 0/1 | |
| | push the orange block onto the red pad | 1/2 | |
| | push the orange block onto the yellow pad | 2/2 | |
| | push the purple block onto the blue pad | 1/1 | |
| | push the purple block onto the green pad | 1/1 | |
| | push the purple block onto the orange pad | 4/5 | |
| | push the purple block onto the purple pad | 0/1 | |
| | push the purple block onto the red pad | 0/2 | |
| | push the purple block onto the yellow pad | 0/1 | |
| | push the red block onto the orange pad | 1/1 | |
| | push the red block onto the purple pad | 0/2 | |
| | push the red block onto the red pad | 0/2 | |
| | push the red block onto the yellow pad | 2/4 | |
| | push the yellow block onto the blue pad | 2/3 | |
| | push the yellow block onto the green pad | 1/3 | |
| | push the yellow block onto the purple pad | 0/1 | |
| | push the yellow block onto the red pad | 0/1 | |
| | push the yellow block onto the yellow pad | 0/1 | |
|
|
| Videos: videos/push_failure_seed1099511627776.mp4, videos/push_success_seed1099511627777.mp4 |
|
|
| ## button: per instruction |
|
|
| | Instruction | Success | |
| | |
| | press the blue button | 3/8 | |
| | press the green button | 3/4 | |
| | press the orange button | 3/5 | |
| | press the purple button | 3/11 | |
| | press the red button | 4/13 | |
| | press the yellow button | 2/9 | |
|
|
| Videos: videos/button_failure_seed1099512676352.mp4, videos/button_success_seed1099512676355.mp4 |
|
|
| ## box: per instruction |
|
|
| | Instruction | Success | |
| | |
| | lift the box and put it on the shelf | 1/50 | |
|
|
| Videos: videos/box_failure_seed1099513724928.mp4, videos/box_success_seed1099513724934.mp4 |
|
|
|
|
