| 2026-08-07 14:17:19,707 INFO MainThread:3125586 [wandb_setup.py:_flush():81] Current SDK version is 0.28.1 |
| 2026-08-07 14:17:19,707 INFO MainThread:3125586 [wandb_setup.py:_flush():81] Configure stats pid to 3125586 |
| 2026-08-07 14:17:19,707 INFO MainThread:3125586 [wandb_setup.py:_flush():81] Loading settings from environment variables |
| 2026-08-07 14:17:19,707 INFO MainThread:3125586 [wandb_init.py:setup_run_log_directory():729] Logging user logs to /mnt/data/experiments/lerobot-train-multi-obj/outputs/wandb/run-20260807_141719-oeh5wr57/logs/debug.log |
| 2026-08-07 14:17:19,707 INFO MainThread:3125586 [wandb_init.py:setup_run_log_directory():730] Logging internal logs to /mnt/data/experiments/lerobot-train-multi-obj/outputs/wandb/run-20260807_141719-oeh5wr57/logs/debug-internal.log |
| 2026-08-07 14:17:19,707 INFO MainThread:3125586 [wandb_init.py:init():772] calling init triggers |
| 2026-08-07 14:17:19,707 INFO MainThread:3125586 [wandb_init.py:init():777] wandb.init called with sweep_config: {} |
| config: {'dataset': {'repo_id': 'Daankrol/pick-and-place-multi-obj', 'root': None, 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': 'main', 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'pi05', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': 'lerobot/pi05_base', 'pretrained_revision': None, 'paligemma_variant': 'gemma_2b', 'action_expert_variant': 'gemma_300m', 'dtype': 'bfloat16', 'chunk_size': 50, 'n_action_steps': 50, 'max_state_dim': 32, 'max_action_dim': 32, 'num_inference_steps': 10, 'time_sampling_beta_alpha': 1.5, 'time_sampling_beta_beta': 1.0, 'time_sampling_scale': 0.999, 'time_sampling_offset': 0.001, 'min_period': 0.004, 'max_period': 4.0, 'use_relative_actions': False, 'relative_exclude_joints': ['gripper'], 'action_feature_names': None, 'rtc_config': None, 'image_resolution': [224, 224], 'empty_cameras': 0, 'tokenizer_max_length': 200, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.QUANTILES: 'QUANTILES'>, 'ACTION': <NormalizationMode.QUANTILES: 'QUANTILES'>}, 'gradient_checkpointing': True, 'compile_model': True, 'compile_mode': 'max-autotune', 'freeze_vision_encoder': False, 'train_expert_only': False, 'optimizer_lr': 2.5e-05, 'optimizer_betas': [0.9, 0.95], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 0.01, 'optimizer_grad_clip_norm': 1.0, 'scheduler_warmup_steps': 1000, 'scheduler_decay_steps': 30000, 'scheduler_decay_lr': 2.5e-06}, 'reward_model': None, 'output_dir': '/mnt/data/experiments/lerobot-train-multi-obj/outputs', 'job_name': 'pi05_pick_and_place', 'resume': False, 'seed': 1000, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 16, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 100000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 25000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 2.5e-05, 'weight_decay': 0.01, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.95], 'eps': 1e-08}, 'scheduler': {'type': 'cosine_decay_with_warmup', 'num_warmup_steps': 1000, 'num_decay_steps': 30000, 'peak_lr': 2.5e-05, 'decay_lr': 2.5e-06}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': False, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}} |
| 2026-08-07 14:17:19,707 INFO MainThread:3125586 [wandb_init.py:init():820] starting backend |
| 2026-08-07 14:17:19,925 INFO MainThread:3125586 [wandb_init.py:init():835] sending inform_init request |
| 2026-08-07 14:17:20,068 INFO MainThread:3125586 [wandb_init.py:init():840] backend started and connected |
| 2026-08-07 14:17:20,072 INFO MainThread:3125586 [wandb_init.py:init():910] updated telemetry |
| 2026-08-07 14:17:20,078 INFO MainThread:3125586 [wandb_init.py:init():933] communicating run to backend with 90.0 second timeout |
| 2026-08-07 14:17:20,755 INFO MainThread:3125586 [wandb_init.py:init():978] starting run threads in backend |
| 2026-08-07 14:17:20,841 INFO MainThread:3125586 [wandb_run.py:_console_start():2621] atexit reg |
| 2026-08-07 14:17:20,842 INFO MainThread:3125586 [wandb_run.py:_redirect():2471] redirect: wrap_raw |
| 2026-08-07 14:17:20,842 INFO MainThread:3125586 [wandb_run.py:_redirect():2540] Wrapping output streams. |
| 2026-08-07 14:17:20,842 INFO MainThread:3125586 [wandb_run.py:_redirect():2563] Redirects installed. |
| 2026-08-07 14:17:20,844 INFO MainThread:3125586 [wandb_init.py:init():1016] run started, returning control to user process |
| 2026-08-08 19:27:00,153 INFO wandb-AsyncioManager-main:3125586 [service_client.py:_forward_responses():122] Reached EOF. |
| 2026-08-08 19:27:00,153 INFO wandb-AsyncioManager-main:3125586 [mailbox.py:close():154] Closing mailbox, abandoning 3 handles. |
| 2026-08-08 19:27:00,154 ERROR wandb-AsyncioManager-main:3125586 [asyncio_manager.py:fn_wrap_exceptions():197] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 195, in fn_wrap_exceptions |
| await fn() |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop |
| await asyncio_compat.race( |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 279, in race |
| async with open_task_group(race=True) as tg: |
| File "/usr/lib/python3.12/contextlib.py", line 217, in __aexit__ |
| await anext(self.gen) |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 241, in open_task_group |
| await task_group._wait_all(race=race, timeout=exit_timeout) |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 181, in _wait_all |
| raise exc |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all |
| result = await handle.wait_async(timeout=None) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async |
| response = await self._handle.wait_async(timeout=timeout) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async |
| raise HandleAbandonedError() |
| wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError |
| 2026-08-08 19:27:01,433 ERROR wandb-AsyncioManager-main:3125586 [asyncio_manager.py:fn_wrap_exceptions():197] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 195, in fn_wrap_exceptions |
| await fn() |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/run_stopping.py", line 87, in loop |
| should_stop = await self._check_should_stop() |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/run_stopping.py", line 115, in _check_should_stop |
| handle = await self._interface.deliver_async(request) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/interface/interface_sock.py", line 58, in deliver_async |
| handle = await self._client.deliver(request) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_client.py", line 57, in deliver |
| handle = self._mailbox.require_response(request) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/devuser/daankrol/physical-ai-studio/library/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox.py", line 89, in require_response |
| raise MailboxClosedError() |
| wandb.sdk.mailbox.mailbox.MailboxClosedError |
|
|
