2024-11-26 22:24:17,299 INFO    MainThread:1599924 [wandb_setup.py:_flush():79] Current SDK version is 0.18.7
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_setup.py:_flush():79] Configure stats pid to 1599924
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_setup.py:_flush():79] Loading settings from /home/pawanprakash/.config/wandb/settings
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_setup.py:_flush():79] Loading settings from /blue/hennig/pawanprakash/diffusion/Diffscp_guide/hydra/singlerun/2024-11-26/finetune_alex20_sccomb/wandb/settings
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_setup.py:_flush():79] Loading settings from environment variables: {'start_method': 'thread', 'root_dir': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/wandb'}
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_setup.py:_flush():79] Applying setup settings: {'mode': 'online', '_disable_service': None}
2024-11-26 22:24:17,300 WARNING MainThread:1599924 [wandb_setup.py:_flush():79] Could not save program above cwd: /blue/hennig/pawanprakash/diffusion/Diffscp_guide/runjobs/guide_alex20/guide_656/./../../../MaterialsDiffusion/diffcsp/run.py
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_setup.py:_flush():79] Inferring run settings from compute environment: {'program_relpath': None, 'program_abspath': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/MaterialsDiffusion/diffcsp/run.py', 'program': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/runjobs/guide_alex20/guide_656/./../../../MaterialsDiffusion/diffcsp/run.py'}
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_setup.py:_flush():79] Applying login settings: {}
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_init.py:_log_setup():533] Logging user logs to ./wandb/run-20241126_222417-csuxafea/logs/debug.log
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_init.py:_log_setup():534] Logging internal logs to ./wandb/run-20241126_222417-csuxafea/logs/debug-internal.log
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_init.py:init():619] calling init triggers
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_init.py:init():626] wandb.init called with sweep_config: {}
config: {}
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_init.py:init():669] starting backend
2024-11-26 22:24:17,300 INFO    MainThread:1599924 [wandb_init.py:init():673] sending inform_init request
2024-11-26 22:24:17,302 INFO    MainThread:1599924 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: fork
2024-11-26 22:24:17,303 INFO    MainThread:1599924 [wandb_init.py:init():686] backend started and connected
2024-11-26 22:24:17,307 INFO    MainThread:1599924 [wandb_init.py:init():781] updated telemetry
2024-11-26 22:24:17,308 INFO    MainThread:1599924 [wandb_init.py:init():814] communicating run to backend with 90.0 second timeout
2024-11-26 22:24:17,975 INFO    MainThread:1599924 [wandb_init.py:init():867] starting run threads in backend
2024-11-26 22:24:18,314 INFO    MainThread:1599924 [wandb_run.py:_console_start():2456] atexit reg
2024-11-26 22:24:18,314 INFO    MainThread:1599924 [wandb_run.py:_redirect():2305] redirect: wrap_raw
2024-11-26 22:24:18,315 INFO    MainThread:1599924 [wandb_run.py:_redirect():2370] Wrapping output streams.
2024-11-26 22:24:18,315 INFO    MainThread:1599924 [wandb_run.py:_redirect():2395] Redirects installed.
2024-11-26 22:24:18,325 INFO    MainThread:1599924 [wandb_init.py:init():911] run started, returning control to user process
2024-11-26 22:24:18,325 INFO    MainThread:1599924 [wandb_watch.py:_watch():71] Watching
2024-11-26 22:24:21,703 INFO    MainThread:1599924 [wandb_run.py:_config_callback():1387] config_cb None None {'data': {'root_path': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/MaterialsDiffusion/data/supccomb_12', 'prop': 'tcad', 'num_targets': 1, 'niggli': True, 'primitive': False, 'graph_method': 'crystalnn', 'lattice_scale_method': 'scale_length', 'preprocess_workers': 30, 'readout': 'mean', 'max_atoms': 12, 'otf_graph': False, 'eval_model_name': 'supccomb12', 'tolerance': 0.1, 'use_space_group': False, 'use_pos_index': False, 'train_max_epochs': 1500, 'early_stopping_patience': 200, 'teacher_forcing_max_epoch': 500, 'datamodule': {'_target_': 'diffcsp.pl_data.datamodule.CrystDataModule', 'datasets': {'train': {'_target_': 'diffcsp.pl_data.dataset.CrystDataset', 'name': 'Formation energy train', 'path': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/MaterialsDiffusion/data/supccomb_12/train.csv', 'save_path': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/MaterialsDiffusion/data/supccomb_12/train_ori.pt', 'prop': 'tcad', 'niggli': True, 'primitive': False, 'graph_method': 'crystalnn', 'tolerance': 0.1, 'use_space_group': False, 'use_pos_index': False, 'lattice_scale_method': 'scale_length', 'preprocess_workers': 30}, 'val': [{'_target_': 'diffcsp.pl_data.dataset.CrystDataset', 'name': 'Formation energy val', 'path': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/MaterialsDiffusion/data/supccomb_12/val.csv', 'save_path': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/MaterialsDiffusion/data/supccomb_12/val_ori.pt', 'prop': 'tcad', 'niggli': True, 'primitive': False, 'graph_method': 'crystalnn', 'tolerance': 0.1, 'use_space_group': False, 'use_pos_index': False, 'lattice_scale_method': 'scale_length', 'preprocess_workers': 30}], 'test': [{'_target_': 'diffcsp.pl_data.dataset.CrystDataset', 'name': 'Formation energy test', 'path': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/MaterialsDiffusion/data/supccomb_12/test.csv', 'save_path': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/MaterialsDiffusion/data/supccomb_12/test_ori.pt', 'prop': 'tcad', 'niggli': True, 'primitive': False, 'graph_method': 'crystalnn', 'tolerance': 0.1, 'use_space_group': False, 'use_pos_index': False, 'lattice_scale_method': 'scale_length', 'preprocess_workers': 30}]}, 'num_workers': {'train': 0, 'val': 0, 'test': 0}, 'batch_size': {'train': 256, 'val': 128, 'test': 128}}}, 'logging': {'val_check_interval': 1, 'wandb': {'name': 'finetune_alex20_sccomb', 'project': 'diffcsp', 'entity': None, 'log_model': True, 'mode': 'online', 'group': 'finetune_alex20_sccomb'}, 'wandb_watch': {'log': 'all', 'log_freq': 500}, 'lr_monitor': {'logging_interval': 'step', 'log_momentum': False}}, 'model': {'decoder': {'_target_': 'diffcsp.pl_modules.cspnet.CSPNet', 'hidden_dim': 512, 'latent_dim': 0, 'max_atoms': 100, 'num_layers': 6, 'act_fn': 'silu', 'dis_emb': 'sin', 'num_freqs': 128, 'edge_style': 'fc', 'max_neighbors': 20, 'cutoff': 7.0, 'ln': True, 'ip': True, 'prop_embed_dim': 64, 'am_hidden_dim': 128}, 'beta_scheduler': {'_target_': 'diffcsp.pl_modules.diff_utils.BetaScheduler', 'timesteps': 1000, 'scheduler_mode': 'cosine'}, 'sigma_scheduler': {'_target_': 'diffcsp.pl_modules.diff_utils.SigmaScheduler', 'timesteps': 1000, 'sigma_begin': 0.005, 'sigma_end': 0.5}, '_target_': 'diffcsp.pl_modules.diffusion_w_type.CSPDiffusion', 'time_dim': 256, 'latent_dim': 0, 'cost_coord': 1.0, 'cost_lattice': 1.0, 'cost_type': 20.0, 'max_neighbors': 20, 'radius': 7.0, 'timesteps': 1000, 'p_uncond': 0.2, 'guide_w': 3.0}, 'optim': {'optimizer': {'_target_': 'torch.optim.Adam', 'lr': 0.001, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0}, 'use_lr_scheduler': True, 'lr_scheduler': {'_target_': 'torch.optim.lr_scheduler.ReduceLROnPlateau', 'factor': 0.6, 'patience': 30, 'min_lr': 0.0001}}, 'train': {'deterministic': True, 'random_seed': 42, 'pl_trainer': {'fast_dev_run': False, 'accelerator': 'auto', 'precision': 32, 'max_epochs': 1500, 'accumulate_grad_batches': 1, 'num_sanity_val_steps': 2, 'gradient_clip_val': 0.5, 'gradient_clip_algorithm': 'value', 'profiler': 'simple'}, 'monitor_metric': 'val_loss', 'monitor_metric_mode': 'min', 'early_stopping': {'patience': 200, 'verbose': True}, 'model_checkpoints': {'save_top_k': 1, 'verbose': False, 'save_last': False}, 'ckpt_path': '/blue/hennig/pawanprakash/diffusion/Diffscp_guide/hydra/singlerun/2024-11-21/alex_20/epoch=656-step=74898.ckpt'}, 'expname': 'finetune_alex20_sccomb', 'core': {'version': '0.0.1', 'tags': ['2024-11-26']}, 'stats/params_total': 14365540, 'stats/params_trainable': 14365540, 'stats/params_not_trainable': 0}
2024-11-26 22:54:56,158 INFO    MainThread:1599924 [wandb_run.py:_finish():2155] finishing run uf_hennig/diffcsp/csuxafea
2024-11-26 22:54:56,159 INFO    MainThread:1599924 [wandb_run.py:_atexit_cleanup():2420] got exitcode: 0
2024-11-26 22:54:56,160 INFO    MainThread:1599924 [wandb_run.py:_restore():2402] restore
2024-11-26 22:54:56,160 INFO    MainThread:1599924 [wandb_run.py:_restore():2408] restore done
2024-11-26 22:54:58,035 INFO    MainThread:1599924 [wandb_run.py:_footer_history_summary_info():3960] rendering history
2024-11-26 22:54:58,036 INFO    MainThread:1599924 [wandb_run.py:_footer_history_summary_info():3992] rendering summary
2024-11-26 22:54:58,041 INFO    MainThread:1599924 [wandb_run.py:_footer_sync_info():3921] logging synced files