File size: 5,409 Bytes
5b7a8c5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | 2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_setup.py:_flush():81] Configure stats pid to 52342
2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_init.py:setup_run_log_directory():723] Logging user logs to adapters_caution/wandb/run-20260621_164319-9xc5iitk/logs/debug.log
2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to adapters_caution/wandb/run-20260621_164319-9xc5iitk/logs/debug-internal.log
2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_init.py:init():766] calling init triggers
2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
config: {'model': 'mlx-community/Llama-3.2-1B-Instruct-bf16', 'train': True, 'data': 'data/mlx_cot', 'fine_tune_type': 'lora', 'optimizer': 'adam', 'mask_prompt': True, 'num_layers': 16, 'batch_size': 2, 'iters': 2500, 'val_batches': 25, 'learning_rate': 5e-05, 'steps_per_report': 50, 'steps_per_eval': 500, 'grad_accumulation_steps': 1, 'resume_adapter_file': None, 'adapter_path': 'adapters_caution', 'save_every': 500, 'test': False, 'test_batches': 500, 'max_seq_length': 2048, 'config': None, 'grad_checkpoint': True, 'clear_cache_threshold': 0, 'report_to': 'wandb', 'project_name': 'text2sql-qwen-coder', 'seed': 0, 'optimizer_config': {'adam': {}, 'adamw': {}, 'muon': {}, 'sgd': {}, 'adafactor': {}}, 'lr_schedule': None, 'lora_parameters': {'rank': 8, 'dropout': 0.0, 'scale': 20.0}, '_wandb': {}}
2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_init.py:init():814] starting backend
2026-06-21 16:43:19,908 INFO MainThread:52342 [wandb_init.py:init():829] sending inform_init request
2026-06-21 16:43:20,975 INFO MainThread:52342 [wandb_init.py:init():834] backend started and connected
2026-06-21 16:43:20,978 INFO MainThread:52342 [wandb_init.py:init():904] updated telemetry
2026-06-21 16:43:20,992 INFO MainThread:52342 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
2026-06-21 16:43:21,645 INFO MainThread:52342 [wandb_init.py:init():972] starting run threads in backend
2026-06-21 16:43:21,751 INFO MainThread:52342 [wandb_run.py:_console_start():2523] atexit reg
2026-06-21 16:43:21,751 INFO MainThread:52342 [wandb_run.py:_redirect():2373] redirect: wrap_raw
2026-06-21 16:43:21,751 INFO MainThread:52342 [wandb_run.py:_redirect():2442] Wrapping output streams.
2026-06-21 16:43:21,751 INFO MainThread:52342 [wandb_run.py:_redirect():2465] Redirects installed.
2026-06-21 16:43:21,752 INFO MainThread:52342 [wandb_init.py:init():1010] run started, returning control to user process
2026-06-21 18:48:07,836 INFO wandb-AsyncioManager-main:52342 [service_client.py:_forward_responses():122] [no run ID] Reached EOF.
2026-06-21 18:48:07,839 INFO wandb-AsyncioManager-main:52342 [mailbox.py:close():154] [no run ID] Closing mailbox, abandoning 2 handles.
2026-06-21 18:48:07,843 ERROR wandb-AsyncioManager-main:52342 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
Traceback (most recent call last):
File "/Users/apple/Documents/work/Kd-project/Insufficient_Tokens/W5_Fine_Tune/.venv/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
await fn()
File "/Users/apple/Documents/work/Kd-project/Insufficient_Tokens/W5_Fine_Tune/.venv/lib/python3.11/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
await asyncio_compat.race(
File "/Users/apple/Documents/work/Kd-project/Insufficient_Tokens/W5_Fine_Tune/.venv/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
async with open_task_group(race=True) as tg:
File "/opt/homebrew/Cellar/python@3.11/3.11.15/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 217, in __aexit__
await anext(self.gen)
File "/Users/apple/Documents/work/Kd-project/Insufficient_Tokens/W5_Fine_Tune/.venv/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
await task_group._wait_all(race=race, timeout=exit_timeout)
File "/Users/apple/Documents/work/Kd-project/Insufficient_Tokens/W5_Fine_Tune/.venv/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
raise exc
File "/Users/apple/Documents/work/Kd-project/Insufficient_Tokens/W5_Fine_Tune/.venv/lib/python3.11/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
result = await handle.wait_async(timeout=None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/apple/Documents/work/Kd-project/Insufficient_Tokens/W5_Fine_Tune/.venv/lib/python3.11/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
response = await self._handle.wait_async(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/apple/Documents/work/Kd-project/Insufficient_Tokens/W5_Fine_Tune/.venv/lib/python3.11/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
raise HandleAbandonedError()
wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError
|