File size: 16,556 Bytes

GL backend: egl; seeds: checked against 926 dataset seeds: no overlap (max training seed 1200300)
/root/work/Isaac-GR00T/.venv/lib/python3.12/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
  check_for_updates()
Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
/root/work/Isaac-GR00T/gr00t/model/modules/dit.py:278: FutureWarning: Accessing config attribute `compute_dtype` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'compute_dtype' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.compute_dtype'.
  embedding_dim=self.inner_dim, compute_dtype=self.compute_dtype
/root/work/Isaac-GR00T/gr00t/model/modules/dit.py:309: FutureWarning: Accessing config attribute `output_dim` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'output_dim' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.output_dim'.
  self.proj_out_2 = nn.Linear(self.inner_dim, self.output_dim)
Total number of DiT parameters:  1091722240
Total number of SelfAttentionTransformer parameters:  201433088

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 44.04it/s]
[push 1/50] OK   'push the yellow block onto the green pad' steps=78 
[push 2/50] OK   'push the yellow block onto the blue pad' steps=104 
[push 3/50] OK   'push the orange block onto the red pad' steps=117 
[push 4/50] OK   'push the green block onto the blue pad' steps=88 
[push 5/50] OK   'push the orange block onto the yellow pad' steps=85 
[push 6/50] OK   'push the blue block onto the orange pad' steps=55 
[push 7/50] OK   'push the blue block onto the blue pad' steps=192 
[push 8/50] OK   'push the yellow block onto the blue pad' steps=160 
[push 9/50] OK   'push the purple block onto the red pad' steps=75 
[push 10/50] OK   'push the purple block onto the yellow pad' steps=88 
[push 11/50] OK   'push the blue block onto the purple pad' steps=76 
[push 12/50] FAIL 'push the red block onto the yellow pad' steps=440 target_short_of_pad
[push 13/50] FAIL 'push the green block onto the purple pad' steps=440 target_short_of_pad
[push 14/50] OK   'push the yellow block onto the yellow pad' steps=75 
[push 15/50] OK   'push the red block onto the yellow pad' steps=54 
[push 16/50] FAIL 'push the red block onto the yellow pad' steps=440 target_not_moved
[push 17/50] OK   'push the blue block onto the red pad' steps=73 
[push 18/50] FAIL 'push the red block onto the purple pad' steps=440 target_short_of_pad
[push 19/50] OK   'push the red block onto the orange pad' steps=82 
[push 20/50] OK   'push the orange block onto the blue pad' steps=84 
[push 21/50] OK   'push the red block onto the red pad' steps=76 
[push 22/50] OK   'push the green block onto the red pad' steps=88 
[push 23/50] OK   'push the purple block onto the orange pad' steps=74 
[push 24/50] OK   'push the orange block onto the red pad' steps=72 
[push 25/50] OK   'push the orange block onto the blue pad' steps=60 
[push 26/50] OK   'push the blue block onto the green pad' steps=55 
[push 27/50] FAIL 'push the purple block onto the orange pad' steps=440 target_short_of_pad
[push 28/50] FAIL 'push the purple block onto the orange pad' steps=440 target_not_moved
[push 29/50] OK   'push the orange block onto the yellow pad' steps=174 
[push 30/50] FAIL 'push the red block onto the red pad' steps=440 target_not_moved
[push 31/50] OK   'push the yellow block onto the red pad' steps=87 
[push 32/50] OK   'push the orange block onto the purple pad' steps=85 
[push 33/50] OK   'push the purple block onto the blue pad' steps=63 
[push 34/50] OK   'push the purple block onto the orange pad' steps=96 
[push 35/50] OK   'push the blue block onto the green pad' steps=74 
[push 36/50] OK   'push the green block onto the green pad' steps=78 
[push 37/50] OK   'push the yellow block onto the blue pad' steps=77 
[push 38/50] OK   'push the yellow block onto the green pad' steps=92 
[push 39/50] OK   'push the green block onto the green pad' steps=76 
[push 40/50] FAIL 'push the yellow block onto the green pad' steps=440 target_short_of_pad
[push 41/50] OK   'push the purple block onto the orange pad' steps=92 
[push 42/50] OK   'push the red block onto the purple pad' steps=95 
[push 43/50] OK   'push the purple block onto the green pad' steps=72 
[push 44/50] OK   'push the purple block onto the purple pad' steps=45 
[push 45/50] OK   'push the yellow block onto the purple pad' steps=97 
[push 46/50] OK   'push the red block onto the yellow pad' steps=65 
[push 47/50] OK   'push the purple block onto the red pad' steps=75 
[push 48/50] OK   'push the blue block onto the orange pad' steps=58 
[push 49/50] OK   'push the blue block onto the yellow pad' steps=110 
[push 50/50] OK   'push the blue block onto the blue pad' steps=156 
[button 1/50] OK   'press the blue button' steps=68 
[button 2/50] OK   'press the orange button' steps=48 
[button 3/50] OK   'press the purple button' steps=53 
[button 4/50] FAIL 'press the orange button' steps=260 no_press
[button 5/50] FAIL 'press the blue button' steps=260 no_press
[button 6/50] FAIL 'press the purple button' steps=260 no_press
[button 7/50] OK   'press the red button' steps=112 
[button 8/50] OK   'press the blue button' steps=49 
[button 9/50] OK   'press the red button' steps=63 
[button 10/50] OK   'press the yellow button' steps=47 
[button 11/50] FAIL 'press the red button' steps=260 no_press
[button 12/50] OK   'press the purple button' steps=75 
[button 13/50] OK   'press the red button' steps=246 
[button 14/50] FAIL 'press the purple button' steps=260 wrong_button_pressed
[button 15/50] FAIL 'press the green button' steps=260 partial_press
[button 16/50] OK   'press the red button' steps=57 
[button 17/50] FAIL 'press the blue button' steps=260 partial_press
[button 18/50] OK   'press the purple button' steps=63 
[button 19/50] FAIL 'press the yellow button' steps=260 wrong_button_pressed
[button 20/50] OK   'press the red button' steps=53 
[button 21/50] OK   'press the red button' steps=52 
[button 22/50] OK   'press the purple button' steps=94 
[button 23/50] FAIL 'press the yellow button' steps=260 partial_press
[button 24/50] OK   'press the green button' steps=53 
[button 25/50] FAIL 'press the blue button' steps=260 no_press
[button 26/50] OK   'press the yellow button' steps=53 
[button 27/50] OK   'press the yellow button' steps=48 
[button 28/50] OK   'press the green button' steps=49 
[button 29/50] OK   'press the yellow button' steps=138 
[button 30/50] OK   'press the blue button' steps=49 
[button 31/50] OK   'press the red button' steps=44 
[button 32/50] FAIL 'press the red button' steps=260 partial_press
[button 33/50] OK   'press the yellow button' steps=56 
[button 34/50] FAIL 'press the purple button' steps=260 no_press
[button 35/50] OK   'press the orange button' steps=82 
[button 36/50] OK   'press the red button' steps=68 
[button 37/50] FAIL 'press the red button' steps=260 partial_press
[button 38/50] OK   'press the purple button' steps=70 
[button 39/50] OK   'press the purple button' steps=50 
[button 40/50] FAIL 'press the red button' steps=260 no_press
[button 41/50] OK   'press the yellow button' steps=53 
[button 42/50] OK   'press the orange button' steps=41 
[button 43/50] OK   'press the blue button' steps=58 
[button 44/50] FAIL 'press the yellow button' steps=260 no_press
[button 45/50] OK   'press the red button' steps=57 
[button 46/50] FAIL 'press the orange button' steps=260 no_press
[button 47/50] OK   'press the purple button' steps=40 
[button 48/50] FAIL 'press the green button' steps=260 no_press
[button 49/50] FAIL 'press the blue button' steps=260 partial_press
[button 50/50] OK   'press the purple button' steps=40 
[box 1/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 2/50] FAIL 'lift the box and put it on the shelf' steps=440 tipped
[box 3/50] OK   'lift the box and put it on the shelf' steps=278 
[box 4/50] OK   'lift the box and put it on the shelf' steps=228 
[box 5/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 6/50] FAIL 'lift the box and put it on the shelf' steps=440 off_shelf_footprint_or_moving
[box 7/50] OK   'lift the box and put it on the shelf' steps=297 
[box 8/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 9/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_off_table
[box 10/50] OK   'lift the box and put it on the shelf' steps=229 
[box 11/50] OK   'lift the box and put it on the shelf' steps=236 
[box 12/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 13/50] OK   'lift the box and put it on the shelf' steps=339 
[box 14/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 15/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 16/50] OK   'lift the box and put it on the shelf' steps=440 
[box 17/50] OK   'lift the box and put it on the shelf' steps=244 
[box 18/50] OK   'lift the box and put it on the shelf' steps=243 
[box 19/50] OK   'lift the box and put it on the shelf' steps=186 
[box 20/50] OK   'lift the box and put it on the shelf' steps=190 
[box 21/50] OK   'lift the box and put it on the shelf' steps=220 
[box 22/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 23/50] OK   'lift the box and put it on the shelf' steps=207 
[box 24/50] OK   'lift the box and put it on the shelf' steps=258 
[box 25/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 26/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 27/50] OK   'lift the box and put it on the shelf' steps=326 
[box 28/50] OK   'lift the box and put it on the shelf' steps=197 
[box 29/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 30/50] OK   'lift the box and put it on the shelf' steps=365 
[box 31/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 32/50] FAIL 'lift the box and put it on the shelf' steps=440 off_shelf_footprint_or_moving
[box 33/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 34/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 35/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 36/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 37/50] OK   'lift the box and put it on the shelf' steps=277 
[box 38/50] FAIL 'lift the box and put it on the shelf' steps=440 not_released
[box 39/50] OK   'lift the box and put it on the shelf' steps=314 
[box 40/50] OK   'lift the box and put it on the shelf' steps=238 
[box 41/50] OK   'lift the box and put it on the shelf' steps=414 
[box 42/50] OK   'lift the box and put it on the shelf' steps=217 
[box 43/50] OK   'lift the box and put it on the shelf' steps=314 
[box 44/50] FAIL 'lift the box and put it on the shelf' steps=440 dropped_or_not_placed
[box 45/50] OK   'lift the box and put it on the shelf' steps=195 
[box 46/50] OK   'lift the box and put it on the shelf' steps=325 
[box 47/50] OK   'lift the box and put it on the shelf' steps=254 
[box 48/50] FAIL 'lift the box and put it on the shelf' steps=440 tipped
[box 49/50] OK   'lift the box and put it on the shelf' steps=230 
[box 50/50] OK   'lift the box and put it on the shelf' steps=405 
# Closed-loop evaluation: groot (/root/work/runs/asimov_n17/checkpoint-10000)

50 held-out episodes per task (seeds 2**40 + task_id * 2**20 + k; checked against 926 dataset seeds: no overlap (max training seed 1200300)), 8 of 16 actions executed per inference at 20 Hz, success held 10 steps, GL egl.

| Task | Success | 95% CI | Mean time to success | Failures |
| --- | --- | --- | --- | --- |
| push | 42/50 (84%) | 0.71-0.92 | 3.93 s | target_short_of_pad 5, target_not_moved 3 |
| button | 32/50 (64%) | 0.50-0.76 | 2.88 s | no_press 10, partial_press 6, wrong_button_pressed 2 |
| box | 28/50 (56%) | 0.42-0.69 | 13.26 s | dropped_or_not_placed 16, tipped 2, off_shelf_footprint_or_moving 2, dropped_off_table 1, not_released 1 |

Policy calls: {'mean': 0.095, 'p50': 0.0943, 'p90': 0.096, 'calls': 3970}

## push: per instruction

| Instruction | Success |
| --- | --- |
| push the blue block onto the blue pad | 2/2 |
| push the blue block onto the green pad | 2/2 |
| push the blue block onto the orange pad | 2/2 |
| push the blue block onto the purple pad | 1/1 |
| push the blue block onto the red pad | 1/1 |
| push the blue block onto the yellow pad | 1/1 |
| push the green block onto the blue pad | 1/1 |
| push the green block onto the green pad | 2/2 |
| push the green block onto the purple pad | 0/1 |
| push the green block onto the red pad | 1/1 |
| push the orange block onto the blue pad | 2/2 |
| push the orange block onto the purple pad | 1/1 |
| push the orange block onto the red pad | 2/2 |
| push the orange block onto the yellow pad | 2/2 |
| push the purple block onto the blue pad | 1/1 |
| push the purple block onto the green pad | 1/1 |
| push the purple block onto the orange pad | 3/5 |
| push the purple block onto the purple pad | 1/1 |
| push the purple block onto the red pad | 2/2 |
| push the purple block onto the yellow pad | 1/1 |
| push the red block onto the orange pad | 1/1 |
| push the red block onto the purple pad | 1/2 |
| push the red block onto the red pad | 1/2 |
| push the red block onto the yellow pad | 2/4 |
| push the yellow block onto the blue pad | 3/3 |
| push the yellow block onto the green pad | 2/3 |
| push the yellow block onto the purple pad | 1/1 |
| push the yellow block onto the red pad | 1/1 |
| push the yellow block onto the yellow pad | 1/1 |

Videos: videos/push_success_seed1099511627776.mp4, videos/push_failure_seed1099511627787.mp4

## button: per instruction

| Instruction | Success |
| --- | --- |
| press the blue button | 4/8 |
| press the green button | 2/4 |
| press the orange button | 3/5 |
| press the purple button | 8/11 |
| press the red button | 9/13 |
| press the yellow button | 6/9 |

Videos: videos/button_success_seed1099512676352.mp4, videos/button_failure_seed1099512676355.mp4

## box: per instruction

| Instruction | Success |
| --- | --- |
| lift the box and put it on the shelf | 28/50 |

Videos: videos/box_failure_seed1099513724928.mp4, videos/box_success_seed1099513724930.mp4