| 2026-09-26 00:36:14,198 INFO MainThread:2363188 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2 | |
| 2026-09-26 00:36:14,198 INFO MainThread:2363188 [wandb_setup.py:_flush():81] Configure stats pid to 2363188 | |
| 2026-09-26 00:36:14,198 INFO MainThread:2363188 [wandb_setup.py:_flush():81] Loading settings from environment variables | |
| 2026-09-26 00:36:14,198 INFO MainThread:2363188 [wandb_init.py:setup_run_log_directory():723] Logging user logs to outputs/train/act_so101_redcube_batch32/wandb/run-20260926_003614-437el5go/logs/debug.log | |
| 2026-09-26 00:36:14,198 INFO MainThread:2363188 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to outputs/train/act_so101_redcube_batch32/wandb/run-20260926_003614-437el5go/logs/debug-internal.log | |
| 2026-09-26 00:36:14,198 INFO MainThread:2363188 [wandb_init.py:init():766] calling init triggers | |
| 2026-09-26 00:36:14,198 INFO MainThread:2363188 [wandb_init.py:init():771] wandb.init called with sweep_config: {} | |
| config: {'dataset': {'repo_id': 'Deviant65/so101_redcube_20260925_154203', 'repo_type': 'dataset', 'root': None, 'episodes': None, 'exclude_episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'act', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': True, 'repo_id': 'Deviant65/act_so101_redcube_batch32', 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 100, 'n_action_steps': 100, 'normalization_mapping': {'VISUAL': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'STATE': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'ACTION': <NormalizationMode.MEAN_STD: 'MEAN_STD'>}, 'vision_backbone': 'resnet18', 'pretrained_backbone_weights': 'ResNet18_Weights.IMAGENET1K_V1', 'replace_final_stride_with_dilation': False, 'pre_norm': False, 'dim_model': 512, 'n_heads': 8, 'dim_feedforward': 3200, 'feedforward_activation': 'relu', 'n_encoder_layers': 4, 'n_decoder_layers': 1, 'use_vae': True, 'latent_dim': 32, 'n_vae_encoder_layers': 4, 'temporal_ensemble_coeff': None, 'dropout': 0.1, 'kl_weight': 10.0, 'optimizer_lr': 1e-05, 'optimizer_weight_decay': 0.0001, 'optimizer_lr_backbone': 1e-05}, 'reward_model': None, 'output_dir': 'outputs/train/act_so101_redcube_batch32', 'job_name': 'act_so101_redcube_batch32', 'resume': False, 'seed': 1000, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 32, 'prefetch_factor': 4, 'persistent_workers': True, 'dataloader_multiprocessing_context': 'spawn', 'steps': 100000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'checkpoint_format': <CheckpointFormat.SAFETENSORS: 'safetensors'>, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 1e-05, 'weight_decay': 0.0001, 'grad_clip_norm': 10.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': None, 'parallelism': {'dp_replicate': 1, 'dp_shard': 1, 'context_parallel': {'ring_degree': 1, 'ulysses_degree': 1}, 'cfg_parallel': 1}, 'accelerator': {'mixed_precision': 'no', 'gradient_accumulation': {'steps': 1}, 'fsdp': {'reshard_after_forward': True, 'wrap_modules': None, 'min_num_params': None, 'cpu_offload': False, 'ignored_modules': None}, 'ddp': {'find_unused_parameters': True, 'gradient_as_bucket_view': False, 'static_graph': False}, 'compile': {'enabled': False, 'backend': 'inductor', 'mode': None, 'regional': True}, 'activation_checkpointing': {'mode': <ActivationCheckpointingMode.NONE: 'none'>}}, 'eval': {'n_episodes': 50, 'batch_size': 14, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'ema': {'enable': False, 'decay': None, 'update_after_step': 0, 'inv_gamma': 1.0, 'power': 0.75, 'min_decay': 0.0, 'max_decay': 0.9999, 'use_for_eval': True}, 'wandb': {'enable': True, 'disable_artifact': False, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}} | |
| 2026-09-26 00:36:14,199 INFO MainThread:2363188 [wandb_init.py:init():814] starting backend | |
| 2026-09-26 00:36:14,416 INFO MainThread:2363188 [wandb_init.py:init():829] sending inform_init request | |
| 2026-09-26 00:36:15,806 INFO MainThread:2363188 [wandb_init.py:init():834] backend started and connected | |
| 2026-09-26 00:36:15,808 INFO MainThread:2363188 [wandb_init.py:init():904] updated telemetry | |
| 2026-09-26 00:36:15,809 INFO MainThread:2363188 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout | |
| 2026-09-26 00:36:17,207 INFO MainThread:2363188 [wandb_init.py:init():972] starting run threads in backend | |
| 2026-09-26 00:36:17,246 INFO MainThread:2363188 [wandb_run.py:_console_start():2523] atexit reg | |
| 2026-09-26 00:36:17,246 INFO MainThread:2363188 [wandb_run.py:_redirect():2373] redirect: wrap_raw | |
| 2026-09-26 00:36:17,246 INFO MainThread:2363188 [wandb_run.py:_redirect():2442] Wrapping output streams. | |
| 2026-09-26 00:36:17,246 INFO MainThread:2363188 [wandb_run.py:_redirect():2465] Redirects installed. | |
| 2026-09-26 00:36:17,249 INFO MainThread:2363188 [wandb_init.py:init():1010] run started, returning control to user process | |
| 2026-09-26 05:27:23,880 INFO wandb-AsyncioManager-main:2363188 [service_client.py:_forward_responses():122] Reached EOF. | |
| 2026-09-26 05:27:23,880 INFO wandb-AsyncioManager-main:2363188 [mailbox.py:close():154] Closing mailbox, abandoning 2 handles. | |
| 2026-09-26 05:27:23,880 ERROR wandb-AsyncioManager-main:2363188 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. | |
| Traceback (most recent call last): | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions | |
| await fn() | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop | |
| await asyncio_compat.race( | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race | |
| async with open_task_group(race=True) as tg: | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/contextlib.py", line 217, in __aexit__ | |
| await anext(self.gen) | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group | |
| await task_group._wait_all(race=race, timeout=exit_timeout) | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all | |
| raise exc | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all | |
| result = await handle.wait_async(timeout=None) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async | |
| response = await self._handle.wait_async(timeout=timeout) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/home/aryan/miniforge3/envs/lerobot/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async | |
| raise HandleAbandonedError() | |
| wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError | |