| INFO:root:Extracted global_step 10000 from checkpoint path |
| /root/work/Isaac-GR00T/.venv/lib/python3.12/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. |
| check_for_updates() |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Total number of DiT parameters: 1091722240 |
| INFO:gr00t.model.gr00t_n1d7.gr00t_n1d7:Using AlternateVLDiT for diffusion model |
| Total number of SelfAttentionTransformer parameters: 201433088 |
|
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 45.57it/s] |
| INFO:root:Using 4 denoising steps |
| INFO:root:Current modality config: |
| {'video': ModalityConfig(delta_indices=[0], modality_keys=['head'], sin_cos_embedding_keys=None, mean_std_embedding_keys=None, action_configs=None), 'state': ModalityConfig(delta_indices=[0], modality_keys=['left_arm', 'right_arm', 'waist', 'neck'], sin_cos_embedding_keys=None, mean_std_embedding_keys=None, action_configs=None), 'action': ModalityConfig(delta_indices=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], modality_keys=['left_arm', 'right_arm', 'waist', 'neck'], sin_cos_embedding_keys=None, mean_std_embedding_keys=None, action_configs=[ActionConfig(rep=<ActionRepresentation.RELATIVE: 'relative'>, type=<ActionType.NON_EEF: 'non_eef'>, format=<ActionFormat.DEFAULT: 'default'>, state_key=None), ActionConfig(rep=<ActionRepresentation.RELATIVE: 'relative'>, type=<ActionType.NON_EEF: 'non_eef'>, format=<ActionFormat.DEFAULT: 'default'>, state_key=None), ActionConfig(rep=<ActionRepresentation.RELATIVE: 'relative'>, type=<ActionType.NON_EEF: 'non_eef'>, format=<ActionFormat.DEFAULT: 'default'>, state_key=None), ActionConfig(rep=<ActionRepresentation.RELATIVE: 'relative'>, type=<ActionType.NON_EEF: 'non_eef'>, format=<ActionFormat.DEFAULT: 'default'>, state_key=None)]), 'language': ModalityConfig(delta_indices=[0], modality_keys=['annotation.human.task_description'], sin_cos_embedding_keys=None, mean_std_embedding_keys=None, action_configs=None)} |
| INFO:root:Dataset length: 900 |
| INFO:root:Running evaluation on trajectories: [0, 300, 600] |
| INFO:root:Running trajectory: 0 |
| INFO:root:Using 143 steps (requested: 300, trajectory length: 143) |
| INFO:root:inferencing at step: 0 |
| INFO:root:inferencing at step: 16 |
| INFO:root:inferencing at step: 32 |
| INFO:root:inferencing at step: 48 |
| INFO:root:inferencing at step: 64 |
| INFO:root:inferencing at step: 80 |
| INFO:root:inferencing at step: 96 |
| INFO:root:inferencing at step: 112 |
| INFO:root:inferencing at step: 128 |
| INFO:root:Unnormalized Action MSE across single traj: 0.00448231864720583 |
| INFO:root:Unnormalized Action MAE across single traj: 0.03181375935673714 |
| INFO:root:state_joints vs time (143, 13) |
| INFO:root:gt_action_joints vs time (143, 13) |
| INFO:root:pred_action_joints vs time (143, 13) |
| INFO:root:MSE for trajectory 0: 0.00448231864720583, MAE: 0.03181375935673714 |
| INFO:root:Running trajectory: 300 |
| INFO:root:Using 108 steps (requested: 300, trajectory length: 108) |
| INFO:root:inferencing at step: 0 |
| INFO:root:inferencing at step: 16 |
| INFO:root:inferencing at step: 32 |
| INFO:root:inferencing at step: 48 |
| INFO:root:inferencing at step: 64 |
| INFO:root:inferencing at step: 80 |
| INFO:root:inferencing at step: 96 |
| INFO:root:Unnormalized Action MSE across single traj: 0.002719493815675378 |
| INFO:root:Unnormalized Action MAE across single traj: 0.0312812440097332 |
| INFO:root:state_joints vs time (108, 13) |
| INFO:root:gt_action_joints vs time (108, 13) |
| INFO:root:pred_action_joints vs time (108, 13) |
| INFO:root:MSE for trajectory 300: 0.002719493815675378, MAE: 0.0312812440097332 |
| INFO:root:Running trajectory: 600 |
| INFO:root:Using 270 steps (requested: 300, trajectory length: 270) |
| INFO:root:inferencing at step: 0 |
| INFO:root:inferencing at step: 16 |
| INFO:root:inferencing at step: 32 |
| INFO:root:inferencing at step: 48 |
| INFO:root:inferencing at step: 64 |
| INFO:root:inferencing at step: 80 |
| INFO:root:inferencing at step: 96 |
| INFO:root:inferencing at step: 112 |
| INFO:root:inferencing at step: 128 |
| INFO:root:inferencing at step: 144 |
| INFO:root:inferencing at step: 160 |
| INFO:root:inferencing at step: 176 |
| INFO:root:inferencing at step: 192 |
| INFO:root:inferencing at step: 208 |
| INFO:root:inferencing at step: 224 |
| INFO:root:inferencing at step: 240 |
| INFO:root:inferencing at step: 256 |
| INFO:root:Unnormalized Action MSE across single traj: 0.001021101139485836 |
| INFO:root:Unnormalized Action MAE across single traj: 0.0182601697742939 |
| INFO:root:state_joints vs time (270, 13) |
| INFO:root:gt_action_joints vs time (270, 13) |
| INFO:root:pred_action_joints vs time (270, 13) |
| INFO:root:MSE for trajectory 600: 0.001021101139485836, MAE: 0.0182601697742939 |
| INFO:root:Average MSE across all trajs: 0.0027409710455685854 |
| INFO:root:Average MAE across all trajs: 0.027118390426039696 |
| INFO:root:Done |
|
|
