config=abl_lact_l24 exp=abl_lact_l24 gpus=0 nproc=1 port=29837 root=/data/models/yglee_lager/c32_256_kit_dl/dl3dv_c32_root images=images_8,images_4 da3=/data/models/depth-anything_DA3-GIANT-1.1/model.safetensors {'module': 'train', 'seed': 0, 'opt': {'lr': 0.0004, 'weight_decay': 0.05, 'num_iter_total': 65001, 'lr_scheduler_iter': 65001, 'warmup_steps': 3000, 'batch_size': 16, 'batch_size_test': 16, 'target_has_input_p': 1.0, 'l2_loss_weight': 1.0, 'perceptual_loss_weight': 0.5, 'grad_clip_norm': 3.0, 'allowed_gradnorm_factor': 5, 'betas': [0.9, 0.95], 'lr_scheduler': 'cosine', 'hold_steps': 0, 'cosine_min_range': 0.0, 'freeze_da3': False, 'print_every': 50}, 'model': {'model_type': 'regressive', 'encoder': 'lact', 'lact_nvs_dir': '/data/models/yglee_lager/LaCT/lact_nvs', 'lact_config': '/data/models/yglee_lager/LaCT/lact_nvs/config/lact_l24_d768_ttt2x.yaml', 'lact_ckpt_blocks': 12, 'lact_compile': True, 'da_model_weights_path': '/data/models/depth-anything_DA3-GIANT-1.1/model.safetensors'}, 'data': {'camera_condition_type': 'da3', 'subdataset_list': [{'name': 'dl3dv', 'view_sampler_range': [10, 20], 'expansion_factor': 1.0, 'equalization_length': 200000}], 'num_cond_views_aug': [2, 6], 'aspect_ratio_range': [1.0, 1.0], 'im_size_hw': [256, 256], 'num_views': 10, 'zero_out_cam_cond_p': 0.0, 'num_tgt_views': 8, 'view_to_sample_prob': ['2,8', '3,2', '4,8', '5,2', '6,8']}, 'test_data': {'dataset_names': ['dl3dv', 'dl3dv', 'dl3dv'], 'dataset_names_log': ['dl3dv_2view', 'dl3dv_4view', 'dl3dv_6view'], 'zero_out_cam_cond_p': [0.0, 0.0, 0.0], 'test_view_indices_paths': ['assets/dl3dv_2v.json', 'assets/dl3dv_4v.json', 'assets/dl3dv_6v.json'], 'num_cond_views': [2, 4, 6], 'camera_condition_type': 'da3'}, 'view_sampler': {'train_method': 'bounded_interp', 'test_method': 'fixed'}, 'log_dir': './output', 'wandb': {'api_key_path': './config/api_keys.yaml', 'enabled': True, 'project': 'abl_enc6', 'entity': 'dudrms514', 'name': None, 'resume': 'allow', 'id': None, 'log_iter': 50, 'vis_iter': 500}, 'eval': {'log_iter': 50, 'eval_at_start': True, 'eval_iter': 5000, 'ckpt_iter': 100000000}, '-c': None, 'config/abl_lact_l24': {'yaml': None}, '-e': None, 'abl_lact_l24': None, 'exp_name': 'abl_lact_l24'} Running setup on rank 0 with world size 1 | distributed init (rank 0): 1, gpu 0 [01:47:47.130574] {'module': 'train', 'seed': 0, 'opt': {'lr': 0.0004, 'weight_decay': 0.05, 'num_iter_total': 65001, 'lr_scheduler_iter': 65001, 'warmup_steps': 3000, 'batch_size': 16, 'batch_size_test': 16, 'target_has_input_p': 1.0, 'l2_loss_weight': 1.0, 'perceptual_loss_weight': 0.5, 'grad_clip_norm': 3.0, 'allowed_gradnorm_factor': 5, 'betas': [0.9, 0.95], 'lr_scheduler': 'cosine', 'hold_steps': 0, 'cosine_min_range': 0.0, 'freeze_da3': False, 'print_every': 50}, 'model': {'model_type': 'regressive', 'encoder': 'lact', 'lact_nvs_dir': '/data/models/yglee_lager/LaCT/lact_nvs', 'lact_config': '/data/models/yglee_lager/LaCT/lact_nvs/config/lact_l24_d768_ttt2x.yaml', 'lact_ckpt_blocks': 12, 'lact_compile': True, 'da_model_weights_path': '/data/models/depth-anything_DA3-GIANT-1.1/model.safetensors'}, 'data': {'camera_condition_type': 'da3', 'subdataset_list': [{'name': 'dl3dv', 'view_sampler_range': [10, 20], 'expansion_factor': 1.0, 'equalization_length': 200000}], 'num_cond_views_aug': [2, 6], 'aspect_ratio_range': [1.0, 1.0], 'im_size_hw': [256, 256], 'num_views': 10, 'zero_out_cam_cond_p': 0.0, 'num_tgt_views': 8, 'view_to_sample_prob': ['2,8', '3,2', '4,8', '5,2', '6,8']}, 'test_data': {'dataset_names': ['dl3dv', 'dl3dv', 'dl3dv'], 'dataset_names_log': ['dl3dv_2view', 'dl3dv_4view', 'dl3dv_6view'], 'zero_out_cam_cond_p': [0.0, 0.0, 0.0], 'test_view_indices_paths': ['assets/dl3dv_2v.json', 'assets/dl3dv_4v.json', 'assets/dl3dv_6v.json'], 'num_cond_views': [2, 4, 6], 'camera_condition_type': 'da3'}, 'view_sampler': {'train_method': 'bounded_interp', 'test_method': 'fixed'}, 'log_dir': './output/abl_lact_l24', 'wandb': {'api_key_path': './config/api_keys.yaml', 'enabled': True, 'project': 'abl_enc6', 'entity': 'dudrms514', 'name': None, 'resume': 'allow', 'id': None, 'log_iter': 50, 'vis_iter': 500}, 'eval': {'log_iter': 50, 'eval_at_start': True, 'eval_iter': 5000, 'ckpt_iter': 100000000}, '-c': None, 'config/abl_lact_l24': {'yaml': None}, '-e': None, 'abl_lact_l24': None, 'exp_name': 'abl_lact_l24', 'rank': 0, 'world_size': 1, 'gpu': 0, 'distributed': True, 'dist_backend': 'nccl'} wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information. wandb: Currently logged in as: dudrms514. Use `wandb login --relogin` to force relogin wandb: Tracking run with wandb version 0.18.7 wandb: Run data is saved locally in ./output/abl_lact_l24/checkpoints/wandb/run-20261113_014747-i2mao1ll wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run abl_lact_l24 wandb: ⭐️ View project at https://wandb.ai/dudrms514/abl_enc6 wandb: 🚀 View run at https://wandb.ai/dudrms514/abl_enc6/runs/i2mao1ll [01:47:48.923264] Backing up 317 files (1.63 MB) [01:47:48.942736] Backup completed: 317/317 files copied in 0.03 seconds /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. warnings.warn( # warn only once [rank0]:[W1113 01:47:49.061474811 ProcessGroupNCCL.cpp:5023] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device. [01:47:54.924519] [model] LaCT /data/models/yglee_lager/LaCT/lact_nvs/config/lact_l24_d768_ttt2x.yaml: 24 blocks, ckpt blocks 12, compile True [01:47:54.949188] Optimizer: AdamW, learning_rate: 0.0004, weight decay: 0.05, betas: [0.9, 0.95] [01:47:54.950139] Total parameters: 312.30M, Trainable parameters: 312.30M [01:47:54.950226] Optimized parameters: ['lact.blocks', 'lact.image_token_decoder', 'lact.input_layernorm', 'lact.input_linear'] [01:47:54.950280] Frozen parameters: [] [01:47:54.950345] Parameters with weight decay: ['module.lact.input_linear.weight', 'module.lact.blocks.0.module_list.0.f.to_qkv.weight', 'module.lact.blocks.0.module_list.0.f.c_proj.weight', 'module.lact.blocks.0.module_list.1.f.w0', 'module.lact.blocks.0.module_list.1.f.w1', 'module.lact.blocks.0.module_list.1.f.w2', 'module.lact.blocks.0.module_list.1.f.to_qkv.weight', 'module.lact.blocks.0.module_list.1.f.c_proj.weight', 'module.lact.blocks.0.module_list.1.f.lr_fc.weight', 'module.lact.blocks.0.module_list.2.f.c_fc.weight', 'module.lact.blocks.0.module_list.2.f.c_proj.weight', 'module.lact.blocks.1.module_list.0.f.to_qkv.weight', 'module.lact.blocks.1.module_list.0.f.c_proj.weight', 'module.lact.blocks.1.module_list.1.f.w0', 'module.lact.blocks.1.module_list.1.f.w1', 'module.lact.blocks.1.module_list.1.f.w2', 'module.lact.blocks.1.module_list.1.f.to_qkv.weight', 'module.lact.blocks.1.module_list.1.f.c_proj.weight', 'module.lact.blocks.1.module_list.1.f.lr_fc.weight', 'module.lact.blocks.1.module_list.2.f.c_fc.weight', 'module.lact.blocks.1.module_list.2.f.c_proj.weight', 'module.lact.blocks.2.module_list.0.f.to_qkv.weight', 'module.lact.blocks.2.module_list.0.f.c_proj.weight', 'module.lact.blocks.2.module_list.1.f.w0', 'module.lact.blocks.2.module_list.1.f.w1', 'module.lact.blocks.2.module_list.1.f.w2', 'module.lact.blocks.2.module_list.1.f.to_qkv.weight', 'module.lact.blocks.2.module_list.1.f.c_proj.weight', 'module.lact.blocks.2.module_list.1.f.lr_fc.weight', 'module.lact.blocks.2.module_list.2.f.c_fc.weight', 'module.lact.blocks.2.module_list.2.f.c_proj.weight', 'module.lact.blocks.3.module_list.0.f.to_qkv.weight', 'module.lact.blocks.3.module_list.0.f.c_proj.weight', 'module.lact.blocks.3.module_list.1.f.w0', 'module.lact.blocks.3.module_list.1.f.w1', 'module.lact.blocks.3.module_list.1.f.w2', 'module.lact.blocks.3.module_list.1.f.to_qkv.weight', 'module.lact.blocks.3.module_list.1.f.c_proj.weight', 'module.lact.blocks.3.module_list.1.f.lr_fc.weight', 'module.lact.blocks.3.module_list.2.f.c_fc.weight', 'module.lact.blocks.3.module_list.2.f.c_proj.weight', 'module.lact.blocks.4.module_list.0.f.to_qkv.weight', 'module.lact.blocks.4.module_list.0.f.c_proj.weight', 'module.lact.blocks.4.module_list.1.f.w0', 'module.lact.blocks.4.module_list.1.f.w1', 'module.lact.blocks.4.module_list.1.f.w2', 'module.lact.blocks.4.module_list.1.f.to_qkv.weight', 'module.lact.blocks.4.module_list.1.f.c_proj.weight', 'module.lact.blocks.4.module_list.1.f.lr_fc.weight', 'module.lact.blocks.4.module_list.2.f.c_fc.weight', 'module.lact.blocks.4.module_list.2.f.c_proj.weight', 'module.lact.blocks.5.module_list.0.f.to_qkv.weight', 'module.lact.blocks.5.module_list.0.f.c_proj.weight', 'module.lact.blocks.5.module_list.1.f.w0', 'module.lact.blocks.5.module_list.1.f.w1', 'module.lact.blocks.5.module_list.1.f.w2', 'module.lact.blocks.5.module_list.1.f.to_qkv.weight', 'module.lact.blocks.5.module_list.1.f.c_proj.weight', 'module.lact.blocks.5.module_list.1.f.lr_fc.weight', 'module.lact.blocks.5.module_list.2.f.c_fc.weight', 'module.lact.blocks.5.module_list.2.f.c_proj.weight', 'module.lact.blocks.6.module_list.0.f.to_qkv.weight', 'module.lact.blocks.6.module_list.0.f.c_proj.weight', 'module.lact.blocks.6.module_list.1.f.w0', 'module.lact.blocks.6.module_list.1.f.w1', 'module.lact.blocks.6.module_list.1.f.w2', 'module.lact.blocks.6.module_list.1.f.to_qkv.weight', 'module.lact.blocks.6.module_list.1.f.c_proj.weight', 'module.lact.blocks.6.module_list.1.f.lr_fc.weight', 'module.lact.blocks.6.module_list.2.f.c_fc.weight', 'module.lact.blocks.6.module_list.2.f.c_proj.weight', 'module.lact.blocks.7.module_list.0.f.to_qkv.weight', 'module.lact.blocks.7.module_list.0.f.c_proj.weight', 'module.lact.blocks.7.module_list.1.f.w0', 'module.lact.blocks.7.module_list.1.f.w1', 'module.lact.blocks.7.module_list.1.f.w2', 'module.lact.blocks.7.module_list.1.f.to_qkv.weight', 'module.lact.blocks.7.module_list.1.f.c_proj.weight', 'module.lact.blocks.7.module_list.1.f.lr_fc.weight', 'module.lact.blocks.7.module_list.2.f.c_fc.weight', 'module.lact.blocks.7.module_list.2.f.c_proj.weight', 'module.lact.blocks.8.module_list.0.f.to_qkv.weight', 'module.lact.blocks.8.module_list.0.f.c_proj.weight', 'module.lact.blocks.8.module_list.1.f.w0', 'module.lact.blocks.8.module_list.1.f.w1', 'module.lact.blocks.8.module_list.1.f.w2', 'module.lact.blocks.8.module_list.1.f.to_qkv.weight', 'module.lact.blocks.8.module_list.1.f.c_proj.weight', 'module.lact.blocks.8.module_list.1.f.lr_fc.weight', 'module.lact.blocks.8.module_list.2.f.c_fc.weight', 'module.lact.blocks.8.module_list.2.f.c_proj.weight', 'module.lact.blocks.9.module_list.0.f.to_qkv.weight', 'module.lact.blocks.9.module_list.0.f.c_proj.weight', 'module.lact.blocks.9.module_list.1.f.w0', 'module.lact.blocks.9.module_list.1.f.w1', 'module.lact.blocks.9.module_list.1.f.w2', 'module.lact.blocks.9.module_list.1.f.to_qkv.weight', 'module.lact.blocks.9.module_list.1.f.c_proj.weight', 'module.lact.blocks.9.module_list.1.f.lr_fc.weight', 'module.lact.blocks.9.module_list.2.f.c_fc.weight', 'module.lact.blocks.9.module_list.2.f.c_proj.weight', 'module.lact.blocks.10.module_list.0.f.to_qkv.weight', 'module.lact.blocks.10.module_list.0.f.c_proj.weight', 'module.lact.blocks.10.module_list.1.f.w0', 'module.lact.blocks.10.module_list.1.f.w1', 'module.lact.blocks.10.module_list.1.f.w2', 'module.lact.blocks.10.module_list.1.f.to_qkv.weight', 'module.lact.blocks.10.module_list.1.f.c_proj.weight', 'module.lact.blocks.10.module_list.1.f.lr_fc.weight', 'module.lact.blocks.10.module_list.2.f.c_fc.weight', 'module.lact.blocks.10.module_list.2.f.c_proj.weight', 'module.lact.blocks.11.module_list.0.f.to_qkv.weight', 'module.lact.blocks.11.module_list.0.f.c_proj.weight', 'module.lact.blocks.11.module_list.1.f.w0', 'module.lact.blocks.11.module_list.1.f.w1', 'module.lact.blocks.11.module_list.1.f.w2', 'module.lact.blocks.11.module_list.1.f.to_qkv.weight', 'module.lact.blocks.11.module_list.1.f.c_proj.weight', 'module.lact.blocks.11.module_list.1.f.lr_fc.weight', 'module.lact.blocks.11.module_list.2.f.c_fc.weight', 'module.lact.blocks.11.module_list.2.f.c_proj.weight', 'module.lact.blocks.12.module_list.0.f.to_qkv.weight', 'module.lact.blocks.12.module_list.0.f.c_proj.weight', 'module.lact.blocks.12.module_list.1.f.w0', 'module.lact.blocks.12.module_list.1.f.w1', 'module.lact.blocks.12.module_list.1.f.w2', 'module.lact.blocks.12.module_list.1.f.to_qkv.weight', 'module.lact.blocks.12.module_list.1.f.c_proj.weight', 'module.lact.blocks.12.module_list.1.f.lr_fc.weight', 'module.lact.blocks.12.module_list.2.f.c_fc.weight', 'module.lact.blocks.12.module_list.2.f.c_proj.weight', 'module.lact.blocks.13.module_list.0.f.to_qkv.weight', 'module.lact.blocks.13.module_list.0.f.c_proj.weight', 'module.lact.blocks.13.module_list.1.f.w0', 'module.lact.blocks.13.module_list.1.f.w1', 'module.lact.blocks.13.module_list.1.f.w2', 'module.lact.blocks.13.module_list.1.f.to_qkv.weight', 'module.lact.blocks.13.module_list.1.f.c_proj.weight', 'module.lact.blocks.13.module_list.1.f.lr_fc.weight', 'module.lact.blocks.13.module_list.2.f.c_fc.weight', 'module.lact.blocks.13.module_list.2.f.c_proj.weight', 'module.lact.blocks.14.module_list.0.f.to_qkv.weight', 'module.lact.blocks.14.module_list.0.f.c_proj.weight', 'module.lact.blocks.14.module_list.1.f.w0', 'module.lact.blocks.14.module_list.1.f.w1', 'module.lact.blocks.14.module_list.1.f.w2', 'module.lact.blocks.14.module_list.1.f.to_qkv.weight', 'module.lact.blocks.14.module_list.1.f.c_proj.weight', 'module.lact.blocks.14.module_list.1.f.lr_fc.weight', 'module.lact.blocks.14.module_list.2.f.c_fc.weight', 'module.lact.blocks.14.module_list.2.f.c_proj.weight', 'module.lact.blocks.15.module_list.0.f.to_qkv.weight', 'module.lact.blocks.15.module_list.0.f.c_proj.weight', 'module.lact.blocks.15.module_list.1.f.w0', 'module.lact.blocks.15.module_list.1.f.w1', 'module.lact.blocks.15.module_list.1.f.w2', 'module.lact.blocks.15.module_list.1.f.to_qkv.weight', 'module.lact.blocks.15.module_list.1.f.c_proj.weight', 'module.lact.blocks.15.module_list.1.f.lr_fc.weight', 'module.lact.blocks.15.module_list.2.f.c_fc.weight', 'module.lact.blocks.15.module_list.2.f.c_proj.weight', 'module.lact.blocks.16.module_list.0.f.to_qkv.weight', 'module.lact.blocks.16.module_list.0.f.c_proj.weight', 'module.lact.blocks.16.module_list.1.f.w0', 'module.lact.blocks.16.module_list.1.f.w1', 'module.lact.blocks.16.module_list.1.f.w2', 'module.lact.blocks.16.module_list.1.f.to_qkv.weight', 'module.lact.blocks.16.module_list.1.f.c_proj.weight', 'module.lact.blocks.16.module_list.1.f.lr_fc.weight', 'module.lact.blocks.16.module_list.2.f.c_fc.weight', 'module.lact.blocks.16.module_list.2.f.c_proj.weight', 'module.lact.blocks.17.module_list.0.f.to_qkv.weight', 'module.lact.blocks.17.module_list.0.f.c_proj.weight', 'module.lact.blocks.17.module_list.1.f.w0', 'module.lact.blocks.17.module_list.1.f.w1', 'module.lact.blocks.17.module_list.1.f.w2', 'module.lact.blocks.17.module_list.1.f.to_qkv.weight', 'module.lact.blocks.17.module_list.1.f.c_proj.weight', 'module.lact.blocks.17.module_list.1.f.lr_fc.weight', 'module.lact.blocks.17.module_list.2.f.c_fc.weight', 'module.lact.blocks.17.module_list.2.f.c_proj.weight', 'module.lact.blocks.18.module_list.0.f.to_qkv.weight', 'module.lact.blocks.18.module_list.0.f.c_proj.weight', 'module.lact.blocks.18.module_list.1.f.w0', 'module.lact.blocks.18.module_list.1.f.w1', 'module.lact.blocks.18.module_list.1.f.w2', 'module.lact.blocks.18.module_list.1.f.to_qkv.weight', 'module.lact.blocks.18.module_list.1.f.c_proj.weight', 'module.lact.blocks.18.module_list.1.f.lr_fc.weight', 'module.lact.blocks.18.module_list.2.f.c_fc.weight', 'module.lact.blocks.18.module_list.2.f.c_proj.weight', 'module.lact.blocks.19.module_list.0.f.to_qkv.weight', 'module.lact.blocks.19.module_list.0.f.c_proj.weight', 'module.lact.blocks.19.module_list.1.f.w0', 'module.lact.blocks.19.module_list.1.f.w1', 'module.lact.blocks.19.module_list.1.f.w2', 'module.lact.blocks.19.module_list.1.f.to_qkv.weight', 'module.lact.blocks.19.module_list.1.f.c_proj.weight', 'module.lact.blocks.19.module_list.1.f.lr_fc.weight', 'module.lact.blocks.19.module_list.2.f.c_fc.weight', 'module.lact.blocks.19.module_list.2.f.c_proj.weight', 'module.lact.blocks.20.module_list.0.f.to_qkv.weight', 'module.lact.blocks.20.module_list.0.f.c_proj.weight', 'module.lact.blocks.20.module_list.1.f.w0', 'module.lact.blocks.20.module_list.1.f.w1', 'module.lact.blocks.20.module_list.1.f.w2', 'module.lact.blocks.20.module_list.1.f.to_qkv.weight', 'module.lact.blocks.20.module_list.1.f.c_proj.weight', 'module.lact.blocks.20.module_list.1.f.lr_fc.weight', 'module.lact.blocks.20.module_list.2.f.c_fc.weight', 'module.lact.blocks.20.module_list.2.f.c_proj.weight', 'module.lact.blocks.21.module_list.0.f.to_qkv.weight', 'module.lact.blocks.21.module_list.0.f.c_proj.weight', 'module.lact.blocks.21.module_list.1.f.w0', 'module.lact.blocks.21.module_list.1.f.w1', 'module.lact.blocks.21.module_list.1.f.w2', 'module.lact.blocks.21.module_list.1.f.to_qkv.weight', 'module.lact.blocks.21.module_list.1.f.c_proj.weight', 'module.lact.blocks.21.module_list.1.f.lr_fc.weight', 'module.lact.blocks.21.module_list.2.f.c_fc.weight', 'module.lact.blocks.21.module_list.2.f.c_proj.weight', 'module.lact.blocks.22.module_list.0.f.to_qkv.weight', 'module.lact.blocks.22.module_list.0.f.c_proj.weight', 'module.lact.blocks.22.module_list.1.f.w0', 'module.lact.blocks.22.module_list.1.f.w1', 'module.lact.blocks.22.module_list.1.f.w2', 'module.lact.blocks.22.module_list.1.f.to_qkv.weight', 'module.lact.blocks.22.module_list.1.f.c_proj.weight', 'module.lact.blocks.22.module_list.1.f.lr_fc.weight', 'module.lact.blocks.22.module_list.2.f.c_fc.weight', 'module.lact.blocks.22.module_list.2.f.c_proj.weight', 'module.lact.blocks.23.module_list.0.f.to_qkv.weight', 'module.lact.blocks.23.module_list.0.f.c_proj.weight', 'module.lact.blocks.23.module_list.1.f.w0', 'module.lact.blocks.23.module_list.1.f.w1', 'module.lact.blocks.23.module_list.1.f.w2', 'module.lact.blocks.23.module_list.1.f.to_qkv.weight', 'module.lact.blocks.23.module_list.1.f.c_proj.weight', 'module.lact.blocks.23.module_list.1.f.lr_fc.weight', 'module.lact.blocks.23.module_list.2.f.c_fc.weight', 'module.lact.blocks.23.module_list.2.f.c_proj.weight', 'module.lact.image_token_decoder.1.weight'] [01:47:54.950416] Parameters without weight decay: ['module.lact.input_layernorm.weight', 'module.lact.blocks.0.module_list.0.ln.weight', 'module.lact.blocks.0.module_list.0.f.q_norm.weight', 'module.lact.blocks.0.module_list.0.f.k_norm.weight', 'module.lact.blocks.0.module_list.1.ln.weight', 'module.lact.blocks.0.module_list.1.f.lr_fc.bias', 'module.lact.blocks.0.module_list.1.f.o_norm.weight', 'module.lact.blocks.0.module_list.2.ln.weight', 'module.lact.blocks.1.module_list.0.ln.weight', 'module.lact.blocks.1.module_list.0.f.q_norm.weight', 'module.lact.blocks.1.module_list.0.f.k_norm.weight', 'module.lact.blocks.1.module_list.1.ln.weight', 'module.lact.blocks.1.module_list.1.f.lr_fc.bias', 'module.lact.blocks.1.module_list.1.f.o_norm.weight', 'module.lact.blocks.1.module_list.2.ln.weight', 'module.lact.blocks.2.module_list.0.ln.weight', 'module.lact.blocks.2.module_list.0.f.q_norm.weight', 'module.lact.blocks.2.module_list.0.f.k_norm.weight', 'module.lact.blocks.2.module_list.1.ln.weight', 'module.lact.blocks.2.module_list.1.f.lr_fc.bias', 'module.lact.blocks.2.module_list.1.f.o_norm.weight', 'module.lact.blocks.2.module_list.2.ln.weight', 'module.lact.blocks.3.module_list.0.ln.weight', 'module.lact.blocks.3.module_list.0.f.q_norm.weight', 'module.lact.blocks.3.module_list.0.f.k_norm.weight', 'module.lact.blocks.3.module_list.1.ln.weight', 'module.lact.blocks.3.module_list.1.f.lr_fc.bias', 'module.lact.blocks.3.module_list.1.f.o_norm.weight', 'module.lact.blocks.3.module_list.2.ln.weight', 'module.lact.blocks.4.module_list.0.ln.weight', 'module.lact.blocks.4.module_list.0.f.q_norm.weight', 'module.lact.blocks.4.module_list.0.f.k_norm.weight', 'module.lact.blocks.4.module_list.1.ln.weight', 'module.lact.blocks.4.module_list.1.f.lr_fc.bias', 'module.lact.blocks.4.module_list.1.f.o_norm.weight', 'module.lact.blocks.4.module_list.2.ln.weight', 'module.lact.blocks.5.module_list.0.ln.weight', 'module.lact.blocks.5.module_list.0.f.q_norm.weight', 'module.lact.blocks.5.module_list.0.f.k_norm.weight', 'module.lact.blocks.5.module_list.1.ln.weight', 'module.lact.blocks.5.module_list.1.f.lr_fc.bias', 'module.lact.blocks.5.module_list.1.f.o_norm.weight', 'module.lact.blocks.5.module_list.2.ln.weight', 'module.lact.blocks.6.module_list.0.ln.weight', 'module.lact.blocks.6.module_list.0.f.q_norm.weight', 'module.lact.blocks.6.module_list.0.f.k_norm.weight', 'module.lact.blocks.6.module_list.1.ln.weight', 'module.lact.blocks.6.module_list.1.f.lr_fc.bias', 'module.lact.blocks.6.module_list.1.f.o_norm.weight', 'module.lact.blocks.6.module_list.2.ln.weight', 'module.lact.blocks.7.module_list.0.ln.weight', 'module.lact.blocks.7.module_list.0.f.q_norm.weight', 'module.lact.blocks.7.module_list.0.f.k_norm.weight', 'module.lact.blocks.7.module_list.1.ln.weight', 'module.lact.blocks.7.module_list.1.f.lr_fc.bias', 'module.lact.blocks.7.module_list.1.f.o_norm.weight', 'module.lact.blocks.7.module_list.2.ln.weight', 'module.lact.blocks.8.module_list.0.ln.weight', 'module.lact.blocks.8.module_list.0.f.q_norm.weight', 'module.lact.blocks.8.module_list.0.f.k_norm.weight', 'module.lact.blocks.8.module_list.1.ln.weight', 'module.lact.blocks.8.module_list.1.f.lr_fc.bias', 'module.lact.blocks.8.module_list.1.f.o_norm.weight', 'module.lact.blocks.8.module_list.2.ln.weight', 'module.lact.blocks.9.module_list.0.ln.weight', 'module.lact.blocks.9.module_list.0.f.q_norm.weight', 'module.lact.blocks.9.module_list.0.f.k_norm.weight', 'module.lact.blocks.9.module_list.1.ln.weight', 'module.lact.blocks.9.module_list.1.f.lr_fc.bias', 'module.lact.blocks.9.module_list.1.f.o_norm.weight', 'module.lact.blocks.9.module_list.2.ln.weight', 'module.lact.blocks.10.module_list.0.ln.weight', 'module.lact.blocks.10.module_list.0.f.q_norm.weight', 'module.lact.blocks.10.module_list.0.f.k_norm.weight', 'module.lact.blocks.10.module_list.1.ln.weight', 'module.lact.blocks.10.module_list.1.f.lr_fc.bias', 'module.lact.blocks.10.module_list.1.f.o_norm.weight', 'module.lact.blocks.10.module_list.2.ln.weight', 'module.lact.blocks.11.module_list.0.ln.weight', 'module.lact.blocks.11.module_list.0.f.q_norm.weight', 'module.lact.blocks.11.module_list.0.f.k_norm.weight', 'module.lact.blocks.11.module_list.1.ln.weight', 'module.lact.blocks.11.module_list.1.f.lr_fc.bias', 'module.lact.blocks.11.module_list.1.f.o_norm.weight', 'module.lact.blocks.11.module_list.2.ln.weight', 'module.lact.blocks.12.module_list.0.ln.weight', 'module.lact.blocks.12.module_list.0.f.q_norm.weight', 'module.lact.blocks.12.module_list.0.f.k_norm.weight', 'module.lact.blocks.12.module_list.1.ln.weight', 'module.lact.blocks.12.module_list.1.f.lr_fc.bias', 'module.lact.blocks.12.module_list.1.f.o_norm.weight', 'module.lact.blocks.12.module_list.2.ln.weight', 'module.lact.blocks.13.module_list.0.ln.weight', 'module.lact.blocks.13.module_list.0.f.q_norm.weight', 'module.lact.blocks.13.module_list.0.f.k_norm.weight', 'module.lact.blocks.13.module_list.1.ln.weight', 'module.lact.blocks.13.module_list.1.f.lr_fc.bias', 'module.lact.blocks.13.module_list.1.f.o_norm.weight', 'module.lact.blocks.13.module_list.2.ln.weight', 'module.lact.blocks.14.module_list.0.ln.weight', 'module.lact.blocks.14.module_list.0.f.q_norm.weight', 'module.lact.blocks.14.module_list.0.f.k_norm.weight', 'module.lact.blocks.14.module_list.1.ln.weight', 'module.lact.blocks.14.module_list.1.f.lr_fc.bias', 'module.lact.blocks.14.module_list.1.f.o_norm.weight', 'module.lact.blocks.14.module_list.2.ln.weight', 'module.lact.blocks.15.module_list.0.ln.weight', 'module.lact.blocks.15.module_list.0.f.q_norm.weight', 'module.lact.blocks.15.module_list.0.f.k_norm.weight', 'module.lact.blocks.15.module_list.1.ln.weight', 'module.lact.blocks.15.module_list.1.f.lr_fc.bias', 'module.lact.blocks.15.module_list.1.f.o_norm.weight', 'module.lact.blocks.15.module_list.2.ln.weight', 'module.lact.blocks.16.module_list.0.ln.weight', 'module.lact.blocks.16.module_list.0.f.q_norm.weight', 'module.lact.blocks.16.module_list.0.f.k_norm.weight', 'module.lact.blocks.16.module_list.1.ln.weight', 'module.lact.blocks.16.module_list.1.f.lr_fc.bias', 'module.lact.blocks.16.module_list.1.f.o_norm.weight', 'module.lact.blocks.16.module_list.2.ln.weight', 'module.lact.blocks.17.module_list.0.ln.weight', 'module.lact.blocks.17.module_list.0.f.q_norm.weight', 'module.lact.blocks.17.module_list.0.f.k_norm.weight', 'module.lact.blocks.17.module_list.1.ln.weight', 'module.lact.blocks.17.module_list.1.f.lr_fc.bias', 'module.lact.blocks.17.module_list.1.f.o_norm.weight', 'module.lact.blocks.17.module_list.2.ln.weight', 'module.lact.blocks.18.module_list.0.ln.weight', 'module.lact.blocks.18.module_list.0.f.q_norm.weight', 'module.lact.blocks.18.module_list.0.f.k_norm.weight', 'module.lact.blocks.18.module_list.1.ln.weight', 'module.lact.blocks.18.module_list.1.f.lr_fc.bias', 'module.lact.blocks.18.module_list.1.f.o_norm.weight', 'module.lact.blocks.18.module_list.2.ln.weight', 'module.lact.blocks.19.module_list.0.ln.weight', 'module.lact.blocks.19.module_list.0.f.q_norm.weight', 'module.lact.blocks.19.module_list.0.f.k_norm.weight', 'module.lact.blocks.19.module_list.1.ln.weight', 'module.lact.blocks.19.module_list.1.f.lr_fc.bias', 'module.lact.blocks.19.module_list.1.f.o_norm.weight', 'module.lact.blocks.19.module_list.2.ln.weight', 'module.lact.blocks.20.module_list.0.ln.weight', 'module.lact.blocks.20.module_list.0.f.q_norm.weight', 'module.lact.blocks.20.module_list.0.f.k_norm.weight', 'module.lact.blocks.20.module_list.1.ln.weight', 'module.lact.blocks.20.module_list.1.f.lr_fc.bias', 'module.lact.blocks.20.module_list.1.f.o_norm.weight', 'module.lact.blocks.20.module_list.2.ln.weight', 'module.lact.blocks.21.module_list.0.ln.weight', 'module.lact.blocks.21.module_list.0.f.q_norm.weight', 'module.lact.blocks.21.module_list.0.f.k_norm.weight', 'module.lact.blocks.21.module_list.1.ln.weight', 'module.lact.blocks.21.module_list.1.f.lr_fc.bias', 'module.lact.blocks.21.module_list.1.f.o_norm.weight', 'module.lact.blocks.21.module_list.2.ln.weight', 'module.lact.blocks.22.module_list.0.ln.weight', 'module.lact.blocks.22.module_list.0.f.q_norm.weight', 'module.lact.blocks.22.module_list.0.f.k_norm.weight', 'module.lact.blocks.22.module_list.1.ln.weight', 'module.lact.blocks.22.module_list.1.f.lr_fc.bias', 'module.lact.blocks.22.module_list.1.f.o_norm.weight', 'module.lact.blocks.22.module_list.2.ln.weight', 'module.lact.blocks.23.module_list.0.ln.weight', 'module.lact.blocks.23.module_list.0.f.q_norm.weight', 'module.lact.blocks.23.module_list.0.f.k_norm.weight', 'module.lact.blocks.23.module_list.1.ln.weight', 'module.lact.blocks.23.module_list.1.f.lr_fc.bias', 'module.lact.blocks.23.module_list.1.f.o_norm.weight', 'module.lact.blocks.23.module_list.2.ln.weight', 'module.lact.image_token_decoder.0.weight'] [01:47:54.962724] Found 9517 sequences [01:47:54.963756] [INIT OK] Loaded dataset dl3dv, with length 9517 [01:47:54.963867] Approximately equalizing dataset lengths... [01:47:54.963929] Dataset 0 has length 9517 [01:47:54.963982] Total length: 199857, including the following datasets: [01:47:54.964039] dl3dv: originally 9517, repeated 21 times to reach 199857 examples [01:47:54.964404] For num_cond_views = 2 batch size = 16.0 per GPU [01:47:54.964547] For num_cond_views = 3 batch size = 10.666666666666666 per GPU [01:47:54.964662] For num_cond_views = 4 batch size = 8.0 per GPU [01:47:54.964778] For num_cond_views = 5 batch size = 6.4 per GPU [01:47:54.964889] For num_cond_views = 6 batch size = 5.333333333333333 per GPU [01:47:54.965264] Building dynamic dataloader with seed: 0 [01:47:54.997716] Rank: 0, World size: 1, Distributed rank: 0 [01:47:54.997892] Worker seed: 1 [01:47:55.011042] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.011210] Worker seed: 2 [01:47:55.019832] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.019980] Worker seed: 3 [01:47:55.029244] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.029395] Worker seed: 4 [01:47:55.037651] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.037791] Worker seed: 5 [01:47:55.048383] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.048547] Worker seed: 6 [01:47:55.056187] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.056345] Worker seed: 7 [01:47:55.064926] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.065066] Worker seed: 8 [01:47:55.073747] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.073889] Worker seed: 9 [01:47:55.084681] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.084856] Worker seed: 10 [01:47:55.092770] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.092928] Worker seed: 11 [01:47:55.102449] Rank: 0, World size: 1, Distributed rank: 0 [01:47:55.102610] Worker seed: 12 [01:47:55.127172] [Train] Starting fresh from iter 0 (no checkpoint) [01:47:55.128400] Training model on rank 0 [01:51:01.980082] Unused params (0): [01:51:02.273757] [Epoch 0] | Iter: 1 | Optim: 1 | Time: 187.14s | LR: 0.000000 | SkipOpt: 0 loss: 0.489901 | l2_loss: 0.073823 | psnr: 11.318066 | perceptual_loss: 0.832156 | norm_perceptual_loss: 11.272268 [01:51:03.757748] Found 140 sequences /usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_fx.py:282: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. warnings.warn( [01:53:32.533733] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] /usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. warnings.warn( /usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights. warnings.warn(msg) [01:53:34.056124] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [01:54:32.630461] Gathering scores from all processes... [01:54:32.632448] Padding local tensor of length 140 to max count: 140 [01:54:32.633247] Padding local tensor of length 140 to max count: 140 [01:54:32.633686] Padding local tensor of length 140 to max count: 140 [01:54:32.634089] Gathering scene names from all processes... [01:54:32.634209] Padding local tensor of length 140 on rank 0 to max count: 140 [01:54:32.635666] Evaluation metrics: [01:54:32.636012] psnr: 11.4217 [01:54:32.636196] ssim: 0.0656 [01:54:32.636354] lpips: 0.7886 [01:54:32.640773] Found 140 sequences [01:54:52.230988] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [01:54:53.608490] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [01:55:46.119561] Gathering scores from all processes... [01:55:46.121305] Padding local tensor of length 140 to max count: 140 [01:55:46.122361] Padding local tensor of length 140 to max count: 140 [01:55:46.123268] Padding local tensor of length 140 to max count: 140 [01:55:46.123928] Gathering scene names from all processes... [01:55:46.124078] Padding local tensor of length 140 on rank 0 to max count: 140 [01:55:46.125574] Evaluation metrics: [01:55:46.125841] psnr: 11.4226 [01:55:46.126096] ssim: 0.0656 [01:55:46.126311] lpips: 0.7888 [01:55:46.135734] Found 140 sequences [01:56:08.967399] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [01:56:10.043686] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [01:57:06.101984] Gathering scores from all processes... [01:57:06.103922] Padding local tensor of length 140 to max count: 140 [01:57:06.105013] Padding local tensor of length 140 to max count: 140 [01:57:06.105557] Padding local tensor of length 140 to max count: 140 [01:57:06.105849] Gathering scene names from all processes... [01:57:06.105990] Padding local tensor of length 140 on rank 0 to max count: 140 [01:57:06.106840] Evaluation metrics: [01:57:06.107046] psnr: 11.4229 [01:57:06.107157] ssim: 0.0656 [01:57:06.107258] lpips: 0.7888 [01:57:06.108515] [Eval][iter 0] dl3dv_2view | psnr: 11.421739 | ssim: 0.065604 | lpips: 0.788644 || dl3dv_4view | psnr: 11.422578 | ssim: 0.065605 | lpips: 0.788819 || dl3dv_6view | psnr: 11.422861 | ssim: 0.065604 | lpips: 0.788792 [01:57:06.111125] [Eval][iter 0] dl3dv_2view | psnr: 11.421739 | ssim: 0.065604 | lpips: 0.788644 || dl3dv_4view | psnr: 11.422578 | ssim: 0.065605 | lpips: 0.788819 || dl3dv_6view | psnr: 11.422861 | ssim: 0.065604 | lpips: 0.788792 [02:06:08.281990] [Epoch 0] | Iter: 2 | Optim: 2 | Time: 542.16s | LR: 0.000000 | SkipOpt: 0 loss: 0.411666 | l2_loss: 0.067365 | psnr: 11.715673 | perceptual_loss: 0.688603 | norm_perceptual_loss: 10.222000 [02:06:09.395028] [Epoch 0] | Iter: 3 | Optim: 3 | Time: 1.10s | LR: 0.000000 | SkipOpt: 0 loss: 0.439604 | l2_loss: 0.064536 | psnr: 11.901990 | perceptual_loss: 0.750137 | norm_perceptual_loss: 11.623565 [02:06:10.478069] [Epoch 0] | Iter: 4 | Optim: 4 | Time: 1.07s | LR: 0.000001 | SkipOpt: 0 loss: 0.468399 | l2_loss: 0.075293 | psnr: 11.232451 | perceptual_loss: 0.786212 | norm_perceptual_loss: 10.442020 [02:06:11.561600] [Epoch 0] | Iter: 5 | Optim: 5 | Time: 1.07s | LR: 0.000001 | SkipOpt: 0 loss: 0.463538 | l2_loss: 0.065684 | psnr: 11.825418 | perceptual_loss: 0.795709 | norm_perceptual_loss: 12.114238 [02:06:12.444342] [Epoch 0] | Iter: 6 | Optim: 6 | Time: 0.87s | LR: 0.000001 | SkipOpt: 0 loss: 0.488807 | l2_loss: 0.082359 | psnr: 10.842877 | perceptual_loss: 0.812895 | norm_perceptual_loss: 9.870115 [02:06:13.537019] [Epoch 0] | Iter: 7 | Optim: 7 | Time: 1.08s | LR: 0.000001 | SkipOpt: 0 loss: 0.424847 | l2_loss: 0.068459 | psnr: 11.645700 | perceptual_loss: 0.712776 | norm_perceptual_loss: 10.411740 [02:06:14.345540] [Epoch 0] | Iter: 8 | Optim: 8 | Time: 0.80s | LR: 0.000001 | SkipOpt: 0 loss: 0.485700 | l2_loss: 0.080613 | psnr: 10.935941 | perceptual_loss: 0.810173 | norm_perceptual_loss: 10.050133 [02:06:15.128392] [Epoch 0] | Iter: 9 | Optim: 9 | Time: 0.77s | LR: 0.000001 | SkipOpt: 0 loss: 0.490016 | l2_loss: 0.075196 | psnr: 11.238054 | perceptual_loss: 0.829640 | norm_perceptual_loss: 11.033040 [02:06:16.219119] [Epoch 0] | Iter: 10 | Optim: 10 | Time: 1.08s | LR: 0.000001 | SkipOpt: 0 loss: 0.458098 | l2_loss: 0.074749 | psnr: 11.263948 | perceptual_loss: 0.766699 | norm_perceptual_loss: 10.256986 [02:06:17.000103] [Epoch 0] | Iter: 11 | Optim: 11 | Time: 0.77s | LR: 0.000001 | SkipOpt: 0 loss: 0.458464 | l2_loss: 0.075871 | psnr: 11.199241 | perceptual_loss: 0.765186 | norm_perceptual_loss: 10.085358 [02:06:18.083144] [Epoch 0] | Iter: 12 | Optim: 12 | Time: 1.07s | LR: 0.000002 | SkipOpt: 0 loss: 0.453661 | l2_loss: 0.078016 | psnr: 11.078142 | perceptual_loss: 0.751290 | norm_perceptual_loss: 9.629900 [02:06:19.165474] [Epoch 0] | Iter: 13 | Optim: 13 | Time: 1.07s | LR: 0.000002 | SkipOpt: 0 loss: 0.444981 | l2_loss: 0.065580 | psnr: 11.832310 | perceptual_loss: 0.758803 | norm_perceptual_loss: 11.570707 [02:06:19.988892] [Epoch 0] | Iter: 14 | Optim: 14 | Time: 0.81s | LR: 0.000002 | SkipOpt: 0 loss: 0.445037 | l2_loss: 0.076967 | psnr: 11.136943 | perceptual_loss: 0.736140 | norm_perceptual_loss: 9.564328 [02:06:22.479541] [Epoch 0] | Iter: 15 | Optim: 15 | Time: 2.48s | LR: 0.000002 | SkipOpt: 0 loss: 0.417127 | l2_loss: 0.067542 | psnr: 11.704235 | perceptual_loss: 0.699169 | norm_perceptual_loss: 10.351553 [02:06:24.393160] [Epoch 0] | Iter: 16 | Optim: 16 | Time: 1.90s | LR: 0.000002 | SkipOpt: 0 loss: 0.409225 | l2_loss: 0.068301 | psnr: 11.655714 | perceptual_loss: 0.681847 | norm_perceptual_loss: 9.982934 [02:06:26.305993] [Epoch 0] | Iter: 17 | Optim: 17 | Time: 1.90s | LR: 0.000002 | SkipOpt: 0 loss: 0.412913 | l2_loss: 0.068318 | psnr: 11.654648 | perceptual_loss: 0.689189 | norm_perceptual_loss: 10.087957 [02:06:27.079649] [Epoch 0] | Iter: 18 | Optim: 18 | Time: 0.76s | LR: 0.000002 | SkipOpt: 0 loss: 0.430083 | l2_loss: 0.080995 | psnr: 10.915421 | perceptual_loss: 0.698177 | norm_perceptual_loss: 8.620004 [02:06:27.841567] [Epoch 0] | Iter: 19 | Optim: 19 | Time: 0.75s | LR: 0.000003 | SkipOpt: 0 loss: 0.365644 | l2_loss: 0.055263 | psnr: 12.575623 | perceptual_loss: 0.620761 | norm_perceptual_loss: 11.232771 [02:06:28.609040] [Epoch 0] | Iter: 20 | Optim: 20 | Time: 0.75s | LR: 0.000003 | SkipOpt: 0 loss: 0.424379 | l2_loss: 0.072474 | psnr: 11.398162 | perceptual_loss: 0.703810 | norm_perceptual_loss: 9.711173 [02:06:29.375707] [Epoch 0] | Iter: 21 | Optim: 21 | Time: 0.75s | LR: 0.000003 | SkipOpt: 0 loss: 0.406916 | l2_loss: 0.065257 | psnr: 11.853752 | perceptual_loss: 0.683319 | norm_perceptual_loss: 10.471260 [02:06:30.144138] [Epoch 0] | Iter: 22 | Optim: 22 | Time: 0.76s | LR: 0.000003 | SkipOpt: 0 loss: 0.337790 | l2_loss: 0.051885 | psnr: 12.849545 | perceptual_loss: 0.571808 | norm_perceptual_loss: 11.020592 [02:06:31.228855] [Epoch 0] | Iter: 23 | Optim: 23 | Time: 1.07s | LR: 0.000003 | SkipOpt: 0 loss: 0.401235 | l2_loss: 0.069143 | psnr: 11.602547 | perceptual_loss: 0.664185 | norm_perceptual_loss: 9.606028 [02:06:32.041150] [Epoch 0] | Iter: 24 | Optim: 24 | Time: 0.80s | LR: 0.000003 | SkipOpt: 0 loss: 0.394060 | l2_loss: 0.071324 | psnr: 11.467636 | perceptual_loss: 0.645472 | norm_perceptual_loss: 9.049842 [02:06:33.952349] [Epoch 0] | Iter: 25 | Optim: 25 | Time: 1.90s | LR: 0.000003 | SkipOpt: 0 loss: 0.391139 | l2_loss: 0.059152 | psnr: 12.280319 | perceptual_loss: 0.663975 | norm_perceptual_loss: 11.224930 [02:06:35.097952] [Epoch 0] | Iter: 26 | Optim: 26 | Time: 1.13s | LR: 0.000003 | SkipOpt: 0 loss: 0.376070 | l2_loss: 0.061210 | psnr: 12.131753 | perceptual_loss: 0.629719 | norm_perceptual_loss: 10.287798 [02:06:37.012473] [Epoch 0] | Iter: 27 | Optim: 27 | Time: 1.90s | LR: 0.000004 | SkipOpt: 0 loss: 0.415230 | l2_loss: 0.072673 | psnr: 11.386271 | perceptual_loss: 0.685115 | norm_perceptual_loss: 9.427364 [02:06:37.928153] [Epoch 0] | Iter: 28 | Optim: 28 | Time: 0.90s | LR: 0.000004 | SkipOpt: 0 loss: 0.318417 | l2_loss: 0.048564 | psnr: 13.136854 | perceptual_loss: 0.539706 | norm_perceptual_loss: 11.113284 [02:06:39.025781] [Epoch 0] | Iter: 29 | Optim: 29 | Time: 1.09s | LR: 0.000004 | SkipOpt: 0 loss: 0.434474 | l2_loss: 0.071117 | psnr: 11.480275 | perceptual_loss: 0.726714 | norm_perceptual_loss: 10.218590 [02:06:40.133197] [Epoch 0] | Iter: 30 | Optim: 30 | Time: 1.09s | LR: 0.000004 | SkipOpt: 0 loss: 0.400860 | l2_loss: 0.072621 | psnr: 11.389395 | perceptual_loss: 0.656479 | norm_perceptual_loss: 9.039829 [02:06:42.047721] [Epoch 0] | Iter: 31 | Optim: 31 | Time: 1.90s | LR: 0.000004 | SkipOpt: 0 loss: 0.370734 | l2_loss: 0.062918 | psnr: 12.012264 | perceptual_loss: 0.615633 | norm_perceptual_loss: 9.784722 [02:06:42.827339] [Epoch 0] | Iter: 32 | Optim: 32 | Time: 0.76s | LR: 0.000004 | SkipOpt: 0 loss: 0.437061 | l2_loss: 0.070866 | psnr: 11.495601 | perceptual_loss: 0.732389 | norm_perceptual_loss: 10.334799 [02:06:43.918327] [Epoch 0] | Iter: 33 | Optim: 33 | Time: 1.08s | LR: 0.000004 | SkipOpt: 0 loss: 0.381998 | l2_loss: 0.061503 | psnr: 12.111029 | perceptual_loss: 0.640989 | norm_perceptual_loss: 10.422059 [02:06:44.998606] [Epoch 0] | Iter: 34 | Optim: 34 | Time: 1.07s | LR: 0.000005 | SkipOpt: 0 loss: 0.389818 | l2_loss: 0.073303 | psnr: 11.348789 | perceptual_loss: 0.633029 | norm_perceptual_loss: 8.635804 [02:06:46.911080] [Epoch 0] | Iter: 35 | Optim: 35 | Time: 1.90s | LR: 0.000005 | SkipOpt: 0 loss: 0.403412 | l2_loss: 0.066771 | psnr: 11.754121 | perceptual_loss: 0.673282 | norm_perceptual_loss: 10.083444 [02:06:48.000555] [Epoch 0] | Iter: 36 | Optim: 36 | Time: 1.08s | LR: 0.000005 | SkipOpt: 0 loss: 0.373048 | l2_loss: 0.057889 | psnr: 12.374038 | perceptual_loss: 0.630318 | norm_perceptual_loss: 10.888381 [02:06:49.081633] [Epoch 0] | Iter: 37 | Optim: 37 | Time: 1.07s | LR: 0.000005 | SkipOpt: 0 loss: 0.379724 | l2_loss: 0.057534 | psnr: 12.400770 | perceptual_loss: 0.644381 | norm_perceptual_loss: 11.200046 [02:06:50.164418] [Epoch 0] | Iter: 38 | Optim: 38 | Time: 1.07s | LR: 0.000005 | SkipOpt: 0 loss: 0.427678 | l2_loss: 0.070223 | psnr: 11.535204 | perceptual_loss: 0.714910 | norm_perceptual_loss: 10.180562 [02:06:50.943880] [Epoch 0] | Iter: 39 | Optim: 39 | Time: 0.76s | LR: 0.000005 | SkipOpt: 0 loss: 0.412172 | l2_loss: 0.081198 | psnr: 10.904561 | perceptual_loss: 0.661948 | norm_perceptual_loss: 8.152297 [02:06:51.814029] [Epoch 0] | Iter: 40 | Optim: 40 | Time: 0.86s | LR: 0.000005 | SkipOpt: 0 loss: 0.360382 | l2_loss: 0.059219 | psnr: 12.275362 | perceptual_loss: 0.602326 | norm_perceptual_loss: 10.171096 [02:06:52.900552] [Epoch 0] | Iter: 41 | Optim: 41 | Time: 1.07s | LR: 0.000005 | SkipOpt: 0 loss: 0.373340 | l2_loss: 0.056445 | psnr: 12.483711 | perceptual_loss: 0.633789 | norm_perceptual_loss: 11.228344 [02:06:53.983201] [Epoch 0] | Iter: 42 | Optim: 42 | Time: 1.07s | LR: 0.000006 | SkipOpt: 0 loss: 0.337387 | l2_loss: 0.051740 | psnr: 12.861748 | perceptual_loss: 0.571294 | norm_perceptual_loss: 11.041656 [02:06:54.882250] [Epoch 0] | Iter: 43 | Optim: 43 | Time: 0.89s | LR: 0.000006 | SkipOpt: 0 loss: 0.375108 | l2_loss: 0.056324 | psnr: 12.493073 | perceptual_loss: 0.637567 | norm_perceptual_loss: 11.319657 [02:06:56.794195] [Epoch 0] | Iter: 44 | Optim: 44 | Time: 1.90s | LR: 0.000006 | SkipOpt: 0 loss: 0.403740 | l2_loss: 0.069804 | psnr: 11.561190 | perceptual_loss: 0.667871 | norm_perceptual_loss: 9.567797 [02:06:57.804799] [Epoch 0] | Iter: 45 | Optim: 45 | Time: 1.00s | LR: 0.000006 | SkipOpt: 0 loss: 0.302453 | l2_loss: 0.043222 | psnr: 13.642987 | perceptual_loss: 0.518463 | norm_perceptual_loss: 11.995456 [02:06:58.711497] [Epoch 0] | Iter: 46 | Optim: 46 | Time: 0.89s | LR: 0.000006 | SkipOpt: 0 loss: 0.381503 | l2_loss: 0.059717 | psnr: 12.239010 | perceptual_loss: 0.643572 | norm_perceptual_loss: 10.777008 [02:07:00.625097] [Epoch 0] | Iter: 47 | Optim: 47 | Time: 1.90s | LR: 0.000006 | SkipOpt: 0 loss: 0.366924 | l2_loss: 0.059334 | psnr: 12.266987 | perceptual_loss: 0.615180 | norm_perceptual_loss: 10.368146 [02:07:02.540403] [Epoch 0] | Iter: 48 | Optim: 48 | Time: 1.90s | LR: 0.000006 | SkipOpt: 0 loss: 0.374405 | l2_loss: 0.065243 | psnr: 11.854638 | perceptual_loss: 0.618324 | norm_perceptual_loss: 9.477193 [02:07:03.878181] [Epoch 0] | Iter: 49 | Optim: 49 | Time: 1.33s | LR: 0.000007 | SkipOpt: 0 loss: 0.378028 | l2_loss: 0.059244 | psnr: 12.273583 | perceptual_loss: 0.637569 | norm_perceptual_loss: 10.761816 [02:07:04.962424] [Epoch 0] | Iter: 50 | Optim: 50 | Time: 1.07s | LR: 0.000007 | SkipOpt: 0 loss: 0.382505 | l2_loss: 0.059960 | psnr: 12.221409 | perceptual_loss: 0.645091 | norm_perceptual_loss: 10.758757 [02:07:06.045547] [Epoch 0] | Iter: 51 | Optim: 51 | Time: 1.07s | LR: 0.000007 | SkipOpt: 0 loss: 0.373924 | l2_loss: 0.052046 | psnr: 12.836131 | perceptual_loss: 0.643756 | norm_perceptual_loss: 12.368990 [02:07:07.144494] [Epoch 0] | Iter: 52 | Optim: 52 | Time: 1.09s | LR: 0.000007 | SkipOpt: 0 loss: 0.382662 | l2_loss: 0.058672 | psnr: 12.315694 | perceptual_loss: 0.647981 | norm_perceptual_loss: 11.044130 [02:07:07.949171] [Epoch 0] | Iter: 53 | Optim: 53 | Time: 0.79s | LR: 0.000007 | SkipOpt: 0 loss: 0.343086 | l2_loss: 0.055790 | psnr: 12.534461 | perceptual_loss: 0.574592 | norm_perceptual_loss: 10.299256 [02:07:09.859012] [Epoch 0] | Iter: 54 | Optim: 54 | Time: 1.90s | LR: 0.000007 | SkipOpt: 0 loss: 0.386167 | l2_loss: 0.068541 | psnr: 11.640511 | perceptual_loss: 0.635253 | norm_perceptual_loss: 9.268248 [02:07:11.771319] [Epoch 0] | Iter: 55 | Optim: 55 | Time: 1.90s | LR: 0.000007 | SkipOpt: 0 loss: 0.380664 | l2_loss: 0.062232 | psnr: 12.059877 | perceptual_loss: 0.636865 | norm_perceptual_loss: 10.233750 [02:07:13.689812] [Epoch 0] | Iter: 56 | Optim: 56 | Time: 1.89s | LR: 0.000007 | SkipOpt: 0 loss: 0.394020 | l2_loss: 0.067248 | psnr: 11.723223 | perceptual_loss: 0.653545 | norm_perceptual_loss: 9.718465 [02:07:14.551439] [Epoch 0] | Iter: 57 | Optim: 57 | Time: 0.85s | LR: 0.000008 | SkipOpt: 0 loss: 0.387148 | l2_loss: 0.061257 | psnr: 12.128426 | perceptual_loss: 0.651781 | norm_perceptual_loss: 10.640066 [02:07:16.459496] [Epoch 0] | Iter: 58 | Optim: 58 | Time: 1.89s | LR: 0.000008 | SkipOpt: 0 loss: 0.341809 | l2_loss: 0.050965 | psnr: 12.927286 | perceptual_loss: 0.581689 | norm_perceptual_loss: 11.413509 [02:07:17.537507] [Epoch 0] | Iter: 59 | Optim: 59 | Time: 1.06s | LR: 0.000008 | SkipOpt: 0 loss: 0.381927 | l2_loss: 0.065529 | psnr: 11.835686 | perceptual_loss: 0.632797 | norm_perceptual_loss: 9.656791 [02:07:19.446011] [Epoch 0] | Iter: 60 | Optim: 60 | Time: 1.89s | LR: 0.000008 | SkipOpt: 0 loss: 0.393800 | l2_loss: 0.057141 | psnr: 12.430511 | perceptual_loss: 0.673317 | norm_perceptual_loss: 11.783403 [02:07:21.357943] [Epoch 0] | Iter: 61 | Optim: 61 | Time: 1.90s | LR: 0.000008 | SkipOpt: 0 loss: 0.376002 | l2_loss: 0.056336 | psnr: 12.492170 | perceptual_loss: 0.639333 | norm_perceptual_loss: 11.348644 [02:07:23.266811] [Epoch 0] | Iter: 62 | Optim: 62 | Time: 1.90s | LR: 0.000008 | SkipOpt: 0 loss: 0.331336 | l2_loss: 0.051257 | psnr: 12.902435 | perceptual_loss: 0.560158 | norm_perceptual_loss: 10.928331 [02:07:24.127647] [Epoch 0] | Iter: 63 | Optim: 63 | Time: 0.85s | LR: 0.000008 | SkipOpt: 0 loss: 0.384051 | l2_loss: 0.058111 | psnr: 12.357393 | perceptual_loss: 0.651879 | norm_perceptual_loss: 11.217756 [02:07:26.035349] [Epoch 0] | Iter: 64 | Optim: 64 | Time: 1.89s | LR: 0.000009 | SkipOpt: 0 loss: 0.389569 | l2_loss: 0.064716 | psnr: 11.889900 | perceptual_loss: 0.649706 | norm_perceptual_loss: 10.039379 [02:07:27.944801] [Epoch 0] | Iter: 65 | Optim: 65 | Time: 1.89s | LR: 0.000009 | SkipOpt: 0 loss: 0.419305 | l2_loss: 0.073723 | psnr: 11.323972 | perceptual_loss: 0.691164 | norm_perceptual_loss: 9.375150 [02:07:29.023592] [Epoch 0] | Iter: 66 | Optim: 66 | Time: 1.06s | LR: 0.000009 | SkipOpt: 0 loss: 0.369225 | l2_loss: 0.061318 | psnr: 12.124117 | perceptual_loss: 0.615813 | norm_perceptual_loss: 10.042933 [02:07:29.783997] [Epoch 0] | Iter: 67 | Optim: 67 | Time: 0.75s | LR: 0.000009 | SkipOpt: 0 loss: 0.393544 | l2_loss: 0.060288 | psnr: 12.197719 | perceptual_loss: 0.666513 | norm_perceptual_loss: 11.055556 [02:07:31.689405] [Epoch 0] | Iter: 68 | Optim: 68 | Time: 1.89s | LR: 0.000009 | SkipOpt: 0 loss: 0.371403 | l2_loss: 0.054451 | psnr: 12.639913 | perceptual_loss: 0.633904 | norm_perceptual_loss: 11.641651 [02:07:32.453533] [Epoch 0] | Iter: 69 | Optim: 69 | Time: 0.75s | LR: 0.000009 | SkipOpt: 0 loss: 0.396672 | l2_loss: 0.063490 | psnr: 11.972976 | perceptual_loss: 0.666365 | norm_perceptual_loss: 10.495664 [02:07:34.362752] [Epoch 0] | Iter: 70 | Optim: 70 | Time: 1.89s | LR: 0.000009 | SkipOpt: 0 loss: 0.381006 | l2_loss: 0.064143 | psnr: 11.928501 | perceptual_loss: 0.633726 | norm_perceptual_loss: 9.879875 [02:07:35.126099] [Epoch 0] | Iter: 71 | Optim: 71 | Time: 0.75s | LR: 0.000009 | SkipOpt: 0 loss: 0.341462 | l2_loss: 0.052089 | psnr: 12.832506 | perceptual_loss: 0.578746 | norm_perceptual_loss: 11.110622 [02:07:36.206501] [Epoch 0] | Iter: 72 | Optim: 72 | Time: 1.07s | LR: 0.000010 | SkipOpt: 0 loss: 0.363888 | l2_loss: 0.058758 | psnr: 12.309307 | perceptual_loss: 0.610259 | norm_perceptual_loss: 10.385909 [02:07:36.967559] [Epoch 0] | Iter: 73 | Optim: 73 | Time: 0.75s | LR: 0.000010 | SkipOpt: 0 loss: 0.362067 | l2_loss: 0.053575 | psnr: 12.710351 | perceptual_loss: 0.616983 | norm_perceptual_loss: 11.516178 [02:07:38.042198] [Epoch 0] | Iter: 74 | Optim: 74 | Time: 1.06s | LR: 0.000010 | SkipOpt: 0 loss: 0.311506 | l2_loss: 0.047455 | psnr: 13.237152 | perceptual_loss: 0.528102 | norm_perceptual_loss: 11.128403 [02:07:38.804489] [Epoch 0] | Iter: 75 | Optim: 75 | Time: 0.75s | LR: 0.000010 | SkipOpt: 0 loss: 0.364961 | l2_loss: 0.061940 | psnr: 12.080316 | perceptual_loss: 0.606043 | norm_perceptual_loss: 9.784410 [02:07:40.711427] [Epoch 0] | Iter: 76 | Optim: 76 | Time: 1.89s | LR: 0.000010 | SkipOpt: 0 loss: 0.380449 | l2_loss: 0.061065 | psnr: 12.142086 | perceptual_loss: 0.638768 | norm_perceptual_loss: 10.460479 [02:07:42.617660] [Epoch 0] | Iter: 77 | Optim: 77 | Time: 1.89s | LR: 0.000010 | SkipOpt: 0 loss: 0.363779 | l2_loss: 0.056028 | psnr: 12.515980 | perceptual_loss: 0.615503 | norm_perceptual_loss: 10.985710 [02:07:44.528326] [Epoch 0] | Iter: 78 | Optim: 78 | Time: 1.90s | LR: 0.000010 | SkipOpt: 0 loss: 0.430139 | l2_loss: 0.071994 | psnr: 11.427019 | perceptual_loss: 0.716290 | norm_perceptual_loss: 9.949250 [02:07:45.808554] [Epoch 0] | Iter: 79 | Optim: 79 | Time: 1.27s | LR: 0.000011 | SkipOpt: 0 loss: 0.389260 | l2_loss: 0.058970 | psnr: 12.293712 | perceptual_loss: 0.660581 | norm_perceptual_loss: 11.202044 [02:07:47.718545] [Epoch 0] | Iter: 80 | Optim: 80 | Time: 1.90s | LR: 0.000011 | SkipOpt: 0 loss: 0.360635 | l2_loss: 0.057028 | psnr: 12.439119 | perceptual_loss: 0.607213 | norm_perceptual_loss: 10.647635 [02:07:48.999158] [Epoch 0] | Iter: 81 | Optim: 81 | Time: 1.27s | LR: 0.000011 | SkipOpt: 0 loss: 0.365447 | l2_loss: 0.057917 | psnr: 12.371931 | perceptual_loss: 0.615060 | norm_perceptual_loss: 10.619654 [02:07:50.082184] [Epoch 0] | Iter: 82 | Optim: 82 | Time: 1.07s | LR: 0.000011 | SkipOpt: 0 loss: 0.390183 | l2_loss: 0.056720 | psnr: 12.462635 | perceptual_loss: 0.666926 | norm_perceptual_loss: 11.758206 [02:07:51.989854] [Epoch 0] | Iter: 83 | Optim: 83 | Time: 1.89s | LR: 0.000011 | SkipOpt: 0 loss: 0.392395 | l2_loss: 0.063574 | psnr: 11.967182 | perceptual_loss: 0.657641 | norm_perceptual_loss: 10.344439 [02:07:52.857207] [Epoch 0] | Iter: 84 | Optim: 84 | Time: 0.85s | LR: 0.000011 | SkipOpt: 0 loss: 0.382646 | l2_loss: 0.056960 | psnr: 12.444309 | perceptual_loss: 0.651371 | norm_perceptual_loss: 11.435612 [02:07:53.941691] [Epoch 0] | Iter: 85 | Optim: 85 | Time: 1.07s | LR: 0.000011 | SkipOpt: 0 loss: 0.394568 | l2_loss: 0.058783 | psnr: 12.307510 | perceptual_loss: 0.671572 | norm_perceptual_loss: 11.424664 [02:07:55.857750] [Epoch 0] | Iter: 86 | Optim: 86 | Time: 1.90s | LR: 0.000011 | SkipOpt: 0 loss: 0.409664 | l2_loss: 0.064742 | psnr: 11.888161 | perceptual_loss: 0.689845 | norm_perceptual_loss: 10.655344 [02:07:56.940283] [Epoch 0] | Iter: 87 | Optim: 87 | Time: 1.07s | LR: 0.000012 | SkipOpt: 0 loss: 0.399112 | l2_loss: 0.072631 | psnr: 11.388773 | perceptual_loss: 0.652961 | norm_perceptual_loss: 8.990094 [02:07:58.849806] [Epoch 0] | Iter: 88 | Optim: 88 | Time: 1.90s | LR: 0.000012 | SkipOpt: 0 loss: 0.381300 | l2_loss: 0.061706 | psnr: 12.096748 | perceptual_loss: 0.639189 | norm_perceptual_loss: 10.358664 [02:07:59.984832] [Epoch 0] | Iter: 89 | Optim: 89 | Time: 1.12s | LR: 0.000012 | SkipOpt: 0 loss: 0.416143 | l2_loss: 0.068743 | psnr: 11.627699 | perceptual_loss: 0.694799 | norm_perceptual_loss: 10.107160 [02:08:00.785256] [Epoch 0] | Iter: 90 | Optim: 90 | Time: 0.78s | LR: 0.000012 | SkipOpt: 0 loss: 0.364507 | l2_loss: 0.047744 | psnr: 13.210829 | perceptual_loss: 0.633526 | norm_perceptual_loss: 13.269283 [02:08:02.060666] [Epoch 0] | Iter: 91 | Optim: 91 | Time: 1.26s | LR: 0.000012 | SkipOpt: 0 loss: 0.397940 | l2_loss: 0.060033 | psnr: 12.216076 | perceptual_loss: 0.675814 | norm_perceptual_loss: 11.257313 [02:08:02.948839] [Epoch 0] | Iter: 92 | Optim: 92 | Time: 0.87s | LR: 0.000012 | SkipOpt: 0 loss: 0.387278 | l2_loss: 0.057970 | psnr: 12.367977 | perceptual_loss: 0.658616 | norm_perceptual_loss: 11.361349 [02:08:03.513462] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [02:08:04.865510] [Epoch 0] | Iter: 93 | Optim: 93 | Time: 1.91s | LR: 0.000012 | SkipOpt: 0 loss: 0.370580 | l2_loss: 0.056218 | psnr: 12.501210 | perceptual_loss: 0.628723 | norm_perceptual_loss: 11.183575 [02:08:05.681482] [Epoch 0] | Iter: 94 | Optim: 94 | Time: 0.80s | LR: 0.000013 | SkipOpt: 0 loss: 0.384515 | l2_loss: 0.062114 | psnr: 12.068119 | perceptual_loss: 0.644803 | norm_perceptual_loss: 10.380991 [02:08:06.961790] [Epoch 0] | Iter: 95 | Optim: 95 | Time: 1.27s | LR: 0.000013 | SkipOpt: 0 loss: 0.415208 | l2_loss: 0.075575 | psnr: 11.216201 | perceptual_loss: 0.679265 | norm_perceptual_loss: 8.987926 [02:08:08.870830] [Epoch 0] | Iter: 96 | Optim: 96 | Time: 1.89s | LR: 0.000013 | SkipOpt: 0 loss: 0.377722 | l2_loss: 0.062065 | psnr: 12.071553 | perceptual_loss: 0.631314 | norm_perceptual_loss: 10.171872 [02:08:09.955151] [Epoch 0] | Iter: 97 | Optim: 97 | Time: 1.07s | LR: 0.000013 | SkipOpt: 0 loss: 0.350529 | l2_loss: 0.049484 | psnr: 13.055361 | perceptual_loss: 0.602090 | norm_perceptual_loss: 12.167382 [02:08:11.866313] [Epoch 0] | Iter: 98 | Optim: 98 | Time: 1.90s | LR: 0.000013 | SkipOpt: 0 loss: 0.377625 | l2_loss: 0.063019 | psnr: 12.005300 | perceptual_loss: 0.629212 | norm_perceptual_loss: 9.984521 [02:08:12.640695] [Epoch 0] | Iter: 99 | Optim: 99 | Time: 0.76s | LR: 0.000013 | SkipOpt: 0 loss: 0.332219 | l2_loss: 0.054951 | psnr: 12.600281 | perceptual_loss: 0.554536 | norm_perceptual_loss: 10.091554 [02:08:14.549886] [Epoch 0] | Iter: 100 | Optim: 100 | Time: 1.89s | LR: 0.000013 | SkipOpt: 0 loss: 0.401915 | l2_loss: 0.066686 | psnr: 11.759682 | perceptual_loss: 0.670459 | norm_perceptual_loss: 10.054029 [02:09:08.667544] [Epoch 0] | Iter: 150 | Optim: 150 | Time: 0.80s | LR: 0.000020 | SkipOpt: 0 loss: 0.388834 | l2_loss: 0.059378 | psnr: 12.263717 | perceptual_loss: 0.658911 | norm_perceptual_loss: 11.096822 [02:10:13.927649] [Epoch 0] | Iter: 200 | Optim: 200 | Time: 1.09s | LR: 0.000027 | SkipOpt: 0 loss: 0.398472 | l2_loss: 0.066604 | psnr: 11.764995 | perceptual_loss: 0.663736 | norm_perceptual_loss: 9.965398 [02:11:19.141411] [Epoch 0] | Iter: 250 | Optim: 250 | Time: 1.06s | LR: 0.000033 | SkipOpt: 0 loss: 0.324728 | l2_loss: 0.047411 | psnr: 13.241230 | perceptual_loss: 0.554635 | norm_perceptual_loss: 11.698499 [02:12:21.501589] [Epoch 0] | Iter: 300 | Optim: 300 | Time: 0.81s | LR: 0.000040 | SkipOpt: 0 loss: 0.363469 | l2_loss: 0.048287 | psnr: 13.161731 | perceptual_loss: 0.630364 | norm_perceptual_loss: 13.054630 [02:13:29.179925] [Epoch 0] | Iter: 350 | Optim: 350 | Time: 0.75s | LR: 0.000047 | SkipOpt: 0 loss: 0.281326 | l2_loss: 0.035505 | psnr: 14.497133 | perceptual_loss: 0.491643 | norm_perceptual_loss: 13.847249 [02:14:28.338446] [Epoch 0] | Iter: 400 | Optim: 400 | Time: 1.89s | LR: 0.000053 | SkipOpt: 0 loss: 0.335680 | l2_loss: 0.044022 | psnr: 13.563335 | perceptual_loss: 0.583316 | norm_perceptual_loss: 13.250664 [02:15:32.437421] [Epoch 0] | Iter: 450 | Optim: 450 | Time: 0.78s | LR: 0.000060 | SkipOpt: 0 loss: 0.353465 | l2_loss: 0.052532 | psnr: 12.795780 | perceptual_loss: 0.601866 | norm_perceptual_loss: 11.457181 [02:16:33.667632] [Epoch 0] | Iter: 500 | Optim: 500 | Time: 1.08s | LR: 0.000067 | SkipOpt: 0 loss: 0.339740 | l2_loss: 0.040944 | psnr: 13.878145 | perceptual_loss: 0.597593 | norm_perceptual_loss: 14.595530 [02:17:37.174127] [Epoch 0] | Iter: 550 | Optim: 550 | Time: 1.27s | LR: 0.000073 | SkipOpt: 0 loss: 0.333439 | l2_loss: 0.038669 | psnr: 14.126334 | perceptual_loss: 0.589540 | norm_perceptual_loss: 15.245671 [02:18:36.434826] [Epoch 0] | Iter: 600 | Optim: 600 | Time: 0.75s | LR: 0.000080 | SkipOpt: 0 loss: 0.344509 | l2_loss: 0.045514 | psnr: 13.418559 | perceptual_loss: 0.597989 | norm_perceptual_loss: 13.138606 [02:19:36.568004] [Epoch 0] | Iter: 650 | Optim: 650 | Time: 1.89s | LR: 0.000087 | SkipOpt: 0 loss: 0.318374 | l2_loss: 0.036491 | psnr: 14.378152 | perceptual_loss: 0.563766 | norm_perceptual_loss: 15.449495 [02:20:41.452329] [Epoch 0] | Iter: 700 | Optim: 700 | Time: 0.79s | LR: 0.000093 | SkipOpt: 0 loss: 0.278460 | l2_loss: 0.029780 | psnr: 15.260777 | perceptual_loss: 0.497361 | norm_perceptual_loss: 16.701262 [02:21:47.750525] [Epoch 0] | Iter: 750 | Optim: 750 | Time: 1.07s | LR: 0.000100 | SkipOpt: 0 loss: 0.335991 | l2_loss: 0.039905 | psnr: 13.989745 | perceptual_loss: 0.592172 | norm_perceptual_loss: 14.839608 [02:22:23.127037] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [02:22:50.327346] [Epoch 0] | Iter: 800 | Optim: 800 | Time: 1.06s | LR: 0.000107 | SkipOpt: 0 loss: 0.301495 | l2_loss: 0.030856 | psnr: 15.106562 | perceptual_loss: 0.541278 | norm_perceptual_loss: 17.541878 [02:23:50.946827] [Epoch 0] | Iter: 850 | Optim: 850 | Time: 1.07s | LR: 0.000113 | SkipOpt: 0 loss: 0.314687 | l2_loss: 0.032466 | psnr: 14.885751 | perceptual_loss: 0.564444 | norm_perceptual_loss: 17.385838 [02:24:46.908418] [Epoch 0] | Iter: 900 | Optim: 900 | Time: 1.07s | LR: 0.000120 | SkipOpt: 0 loss: 0.353149 | l2_loss: 0.043777 | psnr: 13.587523 | perceptual_loss: 0.618743 | norm_perceptual_loss: 14.133918 [02:25:49.691378] [Epoch 0] | Iter: 950 | Optim: 950 | Time: 0.78s | LR: 0.000127 | SkipOpt: 0 loss: 0.303986 | l2_loss: 0.029188 | psnr: 15.347952 | perceptual_loss: 0.549595 | norm_perceptual_loss: 18.829470 [02:26:50.818344] [Epoch 0] | Iter: 1000 | Optim: 1000 | Time: 0.88s | LR: 0.000133 | SkipOpt: 0 loss: 0.346159 | l2_loss: 0.038661 | psnr: 14.127254 | perceptual_loss: 0.614997 | norm_perceptual_loss: 15.907364 [02:26:58.522417] Saved checkpoint at iteration 1000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [02:26:58.523075] Saved latest checkpoint at iteration 1000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [02:28:00.901162] [Epoch 0] | Iter: 1050 | Optim: 1050 | Time: 0.81s | LR: 0.000140 | SkipOpt: 0 loss: 0.256843 | l2_loss: 0.022326 | psnr: 16.511957 | perceptual_loss: 0.469034 | norm_perceptual_loss: 21.008732 [02:29:07.971457] [Epoch 0] | Iter: 1100 | Optim: 1100 | Time: 1.07s | LR: 0.000147 | SkipOpt: 0 loss: 0.357331 | l2_loss: 0.037103 | psnr: 14.305960 | perceptual_loss: 0.640456 | norm_perceptual_loss: 17.261766 [02:30:10.611852] [Epoch 0] | Iter: 1150 | Optim: 1150 | Time: 0.87s | LR: 0.000153 | SkipOpt: 0 loss: 0.359165 | l2_loss: 0.037831 | psnr: 14.221552 | perceptual_loss: 0.642668 | norm_perceptual_loss: 16.987970 [02:31:10.096520] [Epoch 0] | Iter: 1200 | Optim: 1200 | Time: 1.12s | LR: 0.000160 | SkipOpt: 0 loss: 0.294371 | l2_loss: 0.025768 | psnr: 15.889128 | perceptual_loss: 0.537204 | norm_perceptual_loss: 20.847416 [02:32:13.596180] [Epoch 0] | Iter: 1250 | Optim: 1250 | Time: 1.06s | LR: 0.000167 | SkipOpt: 0 loss: 0.316622 | l2_loss: 0.033560 | psnr: 14.741716 | perceptual_loss: 0.566124 | norm_perceptual_loss: 16.868748 [02:33:10.081520] [Epoch 0] | Iter: 1300 | Optim: 1300 | Time: 1.90s | LR: 0.000173 | SkipOpt: 0 loss: 0.291897 | l2_loss: 0.026418 | psnr: 15.781034 | perceptual_loss: 0.530958 | norm_perceptual_loss: 20.098505 [02:34:06.690198] [Epoch 0] | Iter: 1350 | Optim: 1350 | Time: 0.84s | LR: 0.000180 | SkipOpt: 0 loss: 0.299965 | l2_loss: 0.028255 | psnr: 15.488996 | perceptual_loss: 0.543419 | norm_perceptual_loss: 19.232433 [02:35:08.973208] [Epoch 0] | Iter: 1400 | Optim: 1400 | Time: 0.78s | LR: 0.000187 | SkipOpt: 0 loss: 0.282101 | l2_loss: 0.027142 | psnr: 15.663578 | perceptual_loss: 0.509918 | norm_perceptual_loss: 18.787031 [02:36:18.086155] [Epoch 0] | Iter: 1450 | Optim: 1450 | Time: 0.93s | LR: 0.000193 | SkipOpt: 0 loss: 0.294474 | l2_loss: 0.027560 | psnr: 15.597137 | perceptual_loss: 0.533828 | norm_perceptual_loss: 19.369349 [02:37:20.332956] [Epoch 0] | Iter: 1500 | Optim: 1500 | Time: 1.89s | LR: 0.000200 | SkipOpt: 0 loss: 0.301852 | l2_loss: 0.028415 | psnr: 15.464537 | perceptual_loss: 0.546875 | norm_perceptual_loss: 19.246044 [02:38:24.769687] [Epoch 0] | Iter: 1550 | Optim: 1550 | Time: 0.93s | LR: 0.000207 | SkipOpt: 0 loss: 0.255953 | l2_loss: 0.020821 | psnr: 16.815086 | perceptual_loss: 0.470264 | norm_perceptual_loss: 22.586571 [02:39:27.637346] [Epoch 0] | Iter: 1600 | Optim: 1600 | Time: 1.96s | LR: 0.000213 | SkipOpt: 0 loss: 0.306046 | l2_loss: 0.030919 | psnr: 15.097738 | perceptual_loss: 0.550255 | norm_perceptual_loss: 17.796627 [02:40:27.118176] [Epoch 0] | Iter: 1650 | Optim: 1650 | Time: 1.90s | LR: 0.000220 | SkipOpt: 0 loss: 0.285223 | l2_loss: 0.024418 | psnr: 16.122868 | perceptual_loss: 0.521610 | norm_perceptual_loss: 21.361544 [02:41:29.777112] [Epoch 0] | Iter: 1700 | Optim: 1700 | Time: 0.75s | LR: 0.000227 | SkipOpt: 0 loss: 0.334454 | l2_loss: 0.031330 | psnr: 15.040360 | perceptual_loss: 0.606248 | norm_perceptual_loss: 19.350254 [02:42:39.343916] [Epoch 0] | Iter: 1750 | Optim: 1750 | Time: 1.90s | LR: 0.000233 | SkipOpt: 0 loss: 0.284204 | l2_loss: 0.026861 | psnr: 15.708832 | perceptual_loss: 0.514686 | norm_perceptual_loss: 19.161337 [02:43:39.508174] [Epoch 0] | Iter: 1800 | Optim: 1800 | Time: 1.89s | LR: 0.000240 | SkipOpt: 0 loss: 0.305903 | l2_loss: 0.026617 | psnr: 15.748436 | perceptual_loss: 0.558573 | norm_perceptual_loss: 20.985704 [02:44:37.469892] [Epoch 0] | Iter: 1850 | Optim: 1850 | Time: 1.08s | LR: 0.000247 | SkipOpt: 0 loss: 0.315275 | l2_loss: 0.027649 | psnr: 15.583160 | perceptual_loss: 0.575251 | norm_perceptual_loss: 20.805279 [02:45:41.791048] [Epoch 0] | Iter: 1900 | Optim: 1900 | Time: 0.75s | LR: 0.000253 | SkipOpt: 0 loss: 0.303869 | l2_loss: 0.034407 | psnr: 14.633473 | perceptual_loss: 0.538922 | norm_perceptual_loss: 15.662938 [02:46:45.921026] [Epoch 0] | Iter: 1950 | Optim: 1950 | Time: 0.75s | LR: 0.000260 | SkipOpt: 0 loss: 0.282240 | l2_loss: 0.027440 | psnr: 15.616123 | perceptual_loss: 0.509600 | norm_perceptual_loss: 18.571259 [02:47:45.501416] [Epoch 0] | Iter: 2000 | Optim: 2000 | Time: 1.07s | LR: 0.000267 | SkipOpt: 0 loss: 0.310746 | l2_loss: 0.031596 | psnr: 15.003679 | perceptual_loss: 0.558300 | norm_perceptual_loss: 17.669941 [02:47:51.405511] Saved checkpoint at iteration 2000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [02:47:51.406204] Saved latest checkpoint at iteration 2000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [02:48:57.367680] [Epoch 0] | Iter: 2050 | Optim: 2050 | Time: 1.89s | LR: 0.000273 | SkipOpt: 0 loss: 0.288011 | l2_loss: 0.025196 | psnr: 15.986694 | perceptual_loss: 0.525630 | norm_perceptual_loss: 20.861681 [02:49:59.634705] [Epoch 0] | Iter: 2100 | Optim: 2100 | Time: 0.79s | LR: 0.000280 | SkipOpt: 0 loss: 0.301758 | l2_loss: 0.029211 | psnr: 15.344478 | perceptual_loss: 0.545093 | norm_perceptual_loss: 18.660280 [02:50:09.366095] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [02:51:01.106793] [Epoch 0] | Iter: 2150 | Optim: 2150 | Time: 1.08s | LR: 0.000287 | SkipOpt: 0 loss: 0.311709 | l2_loss: 0.025712 | psnr: 15.898704 | perceptual_loss: 0.571994 | norm_perceptual_loss: 22.246508 [02:51:58.660239] [Epoch 0] | Iter: 2200 | Optim: 2200 | Time: 1.90s | LR: 0.000293 | SkipOpt: 0 loss: 0.315788 | l2_loss: 0.028776 | psnr: 15.409736 | perceptual_loss: 0.574025 | norm_perceptual_loss: 19.948231 [02:53:01.294422] [Epoch 0] | Iter: 2250 | Optim: 2250 | Time: 0.85s | LR: 0.000300 | SkipOpt: 0 loss: 0.279796 | l2_loss: 0.022231 | psnr: 16.530493 | perceptual_loss: 0.515130 | norm_perceptual_loss: 23.172150 [02:53:58.585256] [Epoch 0] | Iter: 2300 | Optim: 2300 | Time: 1.07s | LR: 0.000307 | SkipOpt: 0 loss: 0.281754 | l2_loss: 0.023288 | psnr: 16.328751 | perceptual_loss: 0.516933 | norm_perceptual_loss: 22.197775 [02:54:59.461548] [Epoch 0] | Iter: 2350 | Optim: 2350 | Time: 1.90s | LR: 0.000313 | SkipOpt: 0 loss: 0.283356 | l2_loss: 0.023962 | psnr: 16.204788 | perceptual_loss: 0.518788 | norm_perceptual_loss: 21.650522 [02:56:02.890692] [Epoch 0] | Iter: 2400 | Optim: 2400 | Time: 1.07s | LR: 0.000320 | SkipOpt: 0 loss: 0.308953 | l2_loss: 0.025255 | psnr: 15.976601 | perceptual_loss: 0.567398 | norm_perceptual_loss: 22.467125 [02:57:09.333186] [Epoch 0] | Iter: 2450 | Optim: 2450 | Time: 4.05s | LR: 0.000327 | SkipOpt: 0 loss: 0.280112 | l2_loss: 0.022755 | psnr: 16.429287 | perceptual_loss: 0.514714 | norm_perceptual_loss: 22.620119 [02:58:07.995514] [Epoch 0] | Iter: 2500 | Optim: 2500 | Time: 0.85s | LR: 0.000333 | SkipOpt: 0 loss: 0.259376 | l2_loss: 0.019849 | psnr: 17.022511 | perceptual_loss: 0.479054 | norm_perceptual_loss: 24.134336 [02:59:09.657154] [Epoch 0] | Iter: 2550 | Optim: 2550 | Time: 0.77s | LR: 0.000340 | SkipOpt: 0 loss: 0.246730 | l2_loss: 0.016624 | psnr: 17.792561 | perceptual_loss: 0.460211 | norm_perceptual_loss: 27.682985 [03:00:14.908734] [Epoch 0] | Iter: 2600 | Optim: 2600 | Time: 0.98s | LR: 0.000347 | SkipOpt: 0 loss: 0.269382 | l2_loss: 0.019915 | psnr: 17.008183 | perceptual_loss: 0.498933 | norm_perceptual_loss: 25.053051 [03:01:19.845129] [Epoch 0] | Iter: 2650 | Optim: 2650 | Time: 1.90s | LR: 0.000353 | SkipOpt: 0 loss: 0.266382 | l2_loss: 0.023982 | psnr: 16.201138 | perceptual_loss: 0.484801 | norm_perceptual_loss: 20.215145 [03:02:22.543130] [Epoch 0] | Iter: 2700 | Optim: 2700 | Time: 1.94s | LR: 0.000360 | SkipOpt: 0 loss: 0.307266 | l2_loss: 0.026429 | psnr: 15.779179 | perceptual_loss: 0.561675 | norm_perceptual_loss: 21.252134 [03:03:33.486674] [Epoch 0] | Iter: 2750 | Optim: 2750 | Time: 0.86s | LR: 0.000367 | SkipOpt: 0 loss: 0.295027 | l2_loss: 0.030227 | psnr: 15.196119 | perceptual_loss: 0.529600 | norm_perceptual_loss: 17.521048 [03:04:44.517635] [Epoch 0] | Iter: 2800 | Optim: 2800 | Time: 1.53s | LR: 0.000373 | SkipOpt: 0 loss: 0.262723 | l2_loss: 0.017362 | psnr: 17.603895 | perceptual_loss: 0.490721 | norm_perceptual_loss: 28.263391 [03:06:00.261281] [Epoch 0] | Iter: 2850 | Optim: 2850 | Time: 1.10s | LR: 0.000380 | SkipOpt: 0 loss: 0.252615 | l2_loss: 0.017856 | psnr: 17.482235 | perceptual_loss: 0.469518 | norm_perceptual_loss: 26.295130 [03:07:04.343464] [Epoch 0] | Iter: 2900 | Optim: 2900 | Time: 1.56s | LR: 0.000387 | SkipOpt: 0 loss: 0.254115 | l2_loss: 0.018660 | psnr: 17.290863 | perceptual_loss: 0.470910 | norm_perceptual_loss: 25.236229 [03:08:07.744253] [Epoch 0] | Iter: 2950 | Optim: 2950 | Time: 0.84s | LR: 0.000393 | SkipOpt: 0 loss: 0.291318 | l2_loss: 0.023240 | psnr: 16.337666 | perceptual_loss: 0.536156 | norm_perceptual_loss: 23.070555 [03:09:12.245971] [Epoch 0] | Iter: 3000 | Optim: 3000 | Time: 0.85s | LR: 0.000400 | SkipOpt: 0 loss: 0.278394 | l2_loss: 0.021591 | psnr: 16.657335 | perceptual_loss: 0.513606 | norm_perceptual_loss: 23.788298 [03:09:17.653549] Saved checkpoint at iteration 3000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [03:09:17.654032] Saved latest checkpoint at iteration 3000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [03:10:20.089821] [Epoch 0] | Iter: 3050 | Optim: 3050 | Time: 1.07s | LR: 0.000400 | SkipOpt: 0 loss: 0.243896 | l2_loss: 0.020142 | psnr: 16.958948 | perceptual_loss: 0.447508 | norm_perceptual_loss: 22.217514 [03:11:17.386495] [Epoch 0] | Iter: 3100 | Optim: 3100 | Time: 1.89s | LR: 0.000400 | SkipOpt: 0 loss: 0.290286 | l2_loss: 0.026068 | psnr: 15.838952 | perceptual_loss: 0.528437 | norm_perceptual_loss: 20.271605 [03:12:29.433977] [Epoch 0] | Iter: 3150 | Optim: 3150 | Time: 1.31s | LR: 0.000400 | SkipOpt: 0 loss: 0.244640 | l2_loss: 0.015593 | psnr: 18.070765 | perceptual_loss: 0.458095 | norm_perceptual_loss: 29.378681 [03:13:36.721489] [Epoch 0] | Iter: 3200 | Optim: 3200 | Time: 1.09s | LR: 0.000400 | SkipOpt: 0 loss: 0.302916 | l2_loss: 0.027682 | psnr: 15.577948 | perceptual_loss: 0.550468 | norm_perceptual_loss: 19.885044 [03:14:40.580313] [Epoch 0] | Iter: 3250 | Optim: 3250 | Time: 1.90s | LR: 0.000400 | SkipOpt: 0 loss: 0.327204 | l2_loss: 0.031624 | psnr: 14.999855 | perceptual_loss: 0.591160 | norm_perceptual_loss: 18.693481 [03:15:38.290232] [Epoch 0] | Iter: 3300 | Optim: 3300 | Time: 0.88s | LR: 0.000400 | SkipOpt: 0 loss: 0.278175 | l2_loss: 0.019984 | psnr: 16.993212 | perceptual_loss: 0.516383 | norm_perceptual_loss: 25.840044 [03:16:37.260302] [Epoch 0] | Iter: 3350 | Optim: 3350 | Time: 1.07s | LR: 0.000400 | SkipOpt: 0 loss: 0.255372 | l2_loss: 0.019116 | psnr: 17.186085 | perceptual_loss: 0.472513 | norm_perceptual_loss: 24.718510 [03:17:38.041070] [Epoch 0] | Iter: 3400 | Optim: 3400 | Time: 0.75s | LR: 0.000400 | SkipOpt: 0 loss: 0.243504 | l2_loss: 0.018000 | psnr: 17.447342 | perceptual_loss: 0.451009 | norm_perceptual_loss: 25.056423 [03:18:37.722264] [Epoch 0] | Iter: 3450 | Optim: 3450 | Time: 1.90s | LR: 0.000400 | SkipOpt: 0 loss: 0.269817 | l2_loss: 0.024820 | psnr: 16.052053 | perceptual_loss: 0.489994 | norm_perceptual_loss: 19.742245 [03:19:40.542124] [Epoch 0] | Iter: 3500 | Optim: 3500 | Time: 1.07s | LR: 0.000400 | SkipOpt: 0 loss: 0.239880 | l2_loss: 0.016516 | psnr: 17.821011 | perceptual_loss: 0.446729 | norm_perceptual_loss: 27.048641 [03:20:52.978267] [Epoch 0] | Iter: 3550 | Optim: 3550 | Time: 1.91s | LR: 0.000400 | SkipOpt: 0 loss: 0.297265 | l2_loss: 0.022015 | psnr: 16.572727 | perceptual_loss: 0.550499 | norm_perceptual_loss: 25.005163 [03:21:59.808718] [Epoch 0] | Iter: 3600 | Optim: 3600 | Time: 1.18s | LR: 0.000400 | SkipOpt: 0 loss: 0.283490 | l2_loss: 0.020736 | psnr: 16.832836 | perceptual_loss: 0.525508 | norm_perceptual_loss: 25.343306 [03:22:41.995926] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [03:23:10.091916] [Epoch 0] | Iter: 3650 | Optim: 3650 | Time: 1.19s | LR: 0.000400 | SkipOpt: 0 loss: 0.277046 | l2_loss: 0.021108 | psnr: 16.755547 | perceptual_loss: 0.511877 | norm_perceptual_loss: 24.250483 [03:24:18.294724] [Epoch 0] | Iter: 3700 | Optim: 3700 | Time: 0.92s | LR: 0.000400 | SkipOpt: 0 loss: 0.234000 | l2_loss: 0.012270 | psnr: 19.111729 | perceptual_loss: 0.443462 | norm_perceptual_loss: 36.143375 [03:25:30.557613] [Epoch 0] | Iter: 3750 | Optim: 3750 | Time: 1.24s | LR: 0.000400 | SkipOpt: 0 loss: 0.264513 | l2_loss: 0.019427 | psnr: 17.115976 | perceptual_loss: 0.490172 | norm_perceptual_loss: 25.231670 [03:26:39.162677] [Epoch 0] | Iter: 3800 | Optim: 3800 | Time: 1.90s | LR: 0.000400 | SkipOpt: 0 loss: 0.288973 | l2_loss: 0.026440 | psnr: 15.777311 | perceptual_loss: 0.525064 | norm_perceptual_loss: 19.858379 [03:27:50.484249] [Epoch 0] | Iter: 3850 | Optim: 3850 | Time: 0.99s | LR: 0.000400 | SkipOpt: 0 loss: 0.282189 | l2_loss: 0.028049 | psnr: 15.520892 | perceptual_loss: 0.508281 | norm_perceptual_loss: 18.121450 [03:29:07.195283] [Epoch 0] | Iter: 3900 | Optim: 3900 | Time: 2.08s | LR: 0.000400 | SkipOpt: 0 loss: 0.257716 | l2_loss: 0.020003 | psnr: 16.988991 | perceptual_loss: 0.475425 | norm_perceptual_loss: 23.767376 [03:30:21.455251] [Epoch 0] | Iter: 3950 | Optim: 3950 | Time: 1.15s | LR: 0.000400 | SkipOpt: 0 loss: 0.279996 | l2_loss: 0.023190 | psnr: 16.347065 | perceptual_loss: 0.513613 | norm_perceptual_loss: 22.148407 [03:31:29.515276] [Epoch 0] | Iter: 4000 | Optim: 4000 | Time: 1.27s | LR: 0.000400 | SkipOpt: 0 loss: 0.262438 | l2_loss: 0.018903 | psnr: 17.234734 | perceptual_loss: 0.487071 | norm_perceptual_loss: 25.767109 [03:31:35.422477] Saved checkpoint at iteration 4000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [03:31:35.423230] Saved latest checkpoint at iteration 4000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [03:32:42.049996] [Epoch 0] | Iter: 4050 | Optim: 4050 | Time: 1.91s | LR: 0.000400 | SkipOpt: 0 loss: 0.302404 | l2_loss: 0.022204 | psnr: 16.535763 | perceptual_loss: 0.560400 | norm_perceptual_loss: 25.239143 [03:33:44.719943] [Epoch 0] | Iter: 4100 | Optim: 4100 | Time: 1.90s | LR: 0.000400 | SkipOpt: 0 loss: 0.254223 | l2_loss: 0.018955 | psnr: 17.222860 | perceptual_loss: 0.470537 | norm_perceptual_loss: 24.824467 [03:34:49.288059] [Epoch 0] | Iter: 4150 | Optim: 4150 | Time: 1.19s | LR: 0.000400 | SkipOpt: 0 loss: 0.259605 | l2_loss: 0.018576 | psnr: 17.310413 | perceptual_loss: 0.482057 | norm_perceptual_loss: 25.950132 [03:36:00.040092] [Epoch 0] | Iter: 4200 | Optim: 4200 | Time: 1.40s | LR: 0.000400 | SkipOpt: 0 loss: 0.273158 | l2_loss: 0.018847 | psnr: 17.247599 | perceptual_loss: 0.508623 | norm_perceptual_loss: 26.987082 [03:37:08.515694] [Epoch 0] | Iter: 4250 | Optim: 4250 | Time: 1.08s | LR: 0.000400 | SkipOpt: 0 loss: 0.275452 | l2_loss: 0.021943 | psnr: 16.586988 | perceptual_loss: 0.507018 | norm_perceptual_loss: 23.105885 [03:38:06.877433] [Epoch 0] | Iter: 4300 | Optim: 4300 | Time: 1.90s | LR: 0.000400 | SkipOpt: 0 loss: 0.277822 | l2_loss: 0.022464 | psnr: 16.485098 | perceptual_loss: 0.510715 | norm_perceptual_loss: 22.734652 [03:39:06.700333] [Epoch 0] | Iter: 4350 | Optim: 4350 | Time: 1.07s | LR: 0.000400 | SkipOpt: 0 loss: 0.308857 | l2_loss: 0.023132 | psnr: 16.357960 | perceptual_loss: 0.571451 | norm_perceptual_loss: 24.704454 [03:40:09.513622] [Epoch 0] | Iter: 4400 | Optim: 4400 | Time: 0.85s | LR: 0.000399 | SkipOpt: 0 loss: 0.227781 | l2_loss: 0.013723 | psnr: 18.625360 | perceptual_loss: 0.428115 | norm_perceptual_loss: 31.195848 [03:41:20.339208] [Epoch 0] | Iter: 4450 | Optim: 4450 | Time: 1.09s | LR: 0.000399 | SkipOpt: 0 loss: 0.266951 | l2_loss: 0.015761 | psnr: 18.024124 | perceptual_loss: 0.502379 | norm_perceptual_loss: 31.874556 [03:42:17.838464] [Epoch 0] | Iter: 4500 | Optim: 4500 | Time: 1.07s | LR: 0.000399 | SkipOpt: 0 loss: 0.305465 | l2_loss: 0.023035 | psnr: 16.376085 | perceptual_loss: 0.564861 | norm_perceptual_loss: 24.521654 [03:43:19.439571] [Epoch 0] | Iter: 4550 | Optim: 4550 | Time: 1.07s | LR: 0.000399 | SkipOpt: 0 loss: 0.250957 | l2_loss: 0.018564 | psnr: 17.313299 | perceptual_loss: 0.464787 | norm_perceptual_loss: 25.037075 [03:44:20.436623] [Epoch 0] | Iter: 4600 | Optim: 4600 | Time: 1.07s | LR: 0.000399 | SkipOpt: 0 loss: 0.249386 | l2_loss: 0.017544 | psnr: 17.558802 | perceptual_loss: 0.463684 | norm_perceptual_loss: 26.430302 [03:45:23.810267] [Epoch 0] | Iter: 4650 | Optim: 4650 | Time: 0.76s | LR: 0.000399 | SkipOpt: 0 loss: 0.213827 | l2_loss: 0.013689 | psnr: 18.636244 | perceptual_loss: 0.400276 | norm_perceptual_loss: 29.240465 [03:46:34.359438] [Epoch 0] | Iter: 4700 | Optim: 4700 | Time: 1.90s | LR: 0.000399 | SkipOpt: 0 loss: 0.287273 | l2_loss: 0.024531 | psnr: 16.102890 | perceptual_loss: 0.525485 | norm_perceptual_loss: 21.421463 [03:47:37.418389] [Epoch 0] | Iter: 4750 | Optim: 4750 | Time: 1.07s | LR: 0.000399 | SkipOpt: 0 loss: 0.279652 | l2_loss: 0.021119 | psnr: 16.753340 | perceptual_loss: 0.517066 | norm_perceptual_loss: 24.483889 [03:48:43.879517] [Epoch 0] | Iter: 4800 | Optim: 4800 | Time: 1.07s | LR: 0.000399 | SkipOpt: 0 loss: 0.273972 | l2_loss: 0.020579 | psnr: 16.865702 | perceptual_loss: 0.506786 | norm_perceptual_loss: 24.626059 [03:49:45.085994] [Epoch 0] | Iter: 4850 | Optim: 4850 | Time: 0.75s | LR: 0.000399 | SkipOpt: 0 loss: 0.264119 | l2_loss: 0.022163 | psnr: 16.543663 | perceptual_loss: 0.483912 | norm_perceptual_loss: 21.833975 [03:50:42.418926] [Epoch 0] | Iter: 4900 | Optim: 4900 | Time: 1.91s | LR: 0.000399 | SkipOpt: 0 loss: 0.251588 | l2_loss: 0.020642 | psnr: 16.852459 | perceptual_loss: 0.461892 | norm_perceptual_loss: 22.376190 [03:51:44.496137] [Epoch 0] | Iter: 4950 | Optim: 4950 | Time: 0.87s | LR: 0.000399 | SkipOpt: 0 loss: 0.300523 | l2_loss: 0.022919 | psnr: 16.398058 | perceptual_loss: 0.555208 | norm_perceptual_loss: 24.224884 [03:52:54.384798] [Epoch 0] | Iter: 5000 | Optim: 5000 | Time: 0.78s | LR: 0.000399 | SkipOpt: 0 loss: 0.238407 | l2_loss: 0.013122 | psnr: 18.819851 | perceptual_loss: 0.450570 | norm_perceptual_loss: 34.335773 [03:52:56.312248] Found 140 sequences [03:53:18.797280] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] /usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. warnings.warn( /usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights. warnings.warn(msg) [03:53:19.650682] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [03:54:12.074362] Gathering scores from all processes... [03:54:12.075937] Padding local tensor of length 140 to max count: 140 [03:54:12.076616] Padding local tensor of length 140 to max count: 140 [03:54:12.077248] Padding local tensor of length 140 to max count: 140 [03:54:12.077759] Gathering scene names from all processes... [03:54:12.078053] Padding local tensor of length 140 on rank 0 to max count: 140 [03:54:12.080514] Evaluation metrics: [03:54:12.080884] psnr: 16.1436 [03:54:12.081176] ssim: 0.4164 [03:54:12.081260] lpips: 0.6651 [03:54:12.087617] Found 140 sequences [03:54:36.384726] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [03:54:37.226209] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [03:55:31.022369] Gathering scores from all processes... [03:55:31.023275] Padding local tensor of length 140 to max count: 140 [03:55:31.023949] Padding local tensor of length 140 to max count: 140 [03:55:31.024512] Padding local tensor of length 140 to max count: 140 [03:55:31.024835] Gathering scene names from all processes... [03:55:31.024985] Padding local tensor of length 140 on rank 0 to max count: 140 [03:55:31.026031] Evaluation metrics: [03:55:31.026241] psnr: 17.6738 [03:55:31.026353] ssim: 0.4401 [03:55:31.026455] lpips: 0.6423 [03:55:31.030255] Found 140 sequences [03:55:57.630503] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [03:55:58.912814] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [03:56:56.330766] Gathering scores from all processes... [03:56:56.332769] Padding local tensor of length 140 to max count: 140 [03:56:56.334268] Padding local tensor of length 140 to max count: 140 [03:56:56.335020] Padding local tensor of length 140 to max count: 140 [03:56:56.335323] Gathering scene names from all processes... [03:56:56.335458] Padding local tensor of length 140 on rank 0 to max count: 140 [03:56:56.336562] Evaluation metrics: [03:56:56.336759] psnr: 18.2426 [03:56:56.336933] ssim: 0.4480 [03:56:56.337166] lpips: 0.6365 [03:56:56.339886] [Eval][iter 5000] dl3dv_2view | psnr: 16.143566 | ssim: 0.416415 | lpips: 0.665120 || dl3dv_4view | psnr: 17.673761 | ssim: 0.440066 | lpips: 0.642324 || dl3dv_6view | psnr: 18.242579 | ssim: 0.447980 | lpips: 0.636527 [03:56:56.345489] [Eval][iter 5000] dl3dv_2view | psnr: 16.143566 | ssim: 0.416415 | lpips: 0.665120 || dl3dv_4view | psnr: 17.673761 | ssim: 0.440066 | lpips: 0.642324 || dl3dv_6view | psnr: 18.242579 | ssim: 0.447980 | lpips: 0.636527 [03:57:01.880078] Saved checkpoint at iteration 5000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [03:57:01.880651] Saved latest checkpoint at iteration 5000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [03:58:07.793360] [Epoch 0] | Iter: 5050 | Optim: 5050 | Time: 1.07s | LR: 0.000399 | SkipOpt: 0 loss: 0.243461 | l2_loss: 0.015204 | psnr: 18.180340 | perceptual_loss: 0.456514 | norm_perceptual_loss: 30.025354 [03:59:05.381734] [Epoch 0] | Iter: 5100 | Optim: 5100 | Time: 1.26s | LR: 0.000399 | SkipOpt: 0 loss: 0.249007 | l2_loss: 0.020672 | psnr: 16.846214 | perceptual_loss: 0.456670 | norm_perceptual_loss: 22.091433 [04:00:11.377302] [Epoch 0] | Iter: 5150 | Optim: 5150 | Time: 1.90s | LR: 0.000399 | SkipOpt: 0 loss: 0.260004 | l2_loss: 0.017682 | psnr: 17.524572 | perceptual_loss: 0.484643 | norm_perceptual_loss: 27.408129 [04:01:19.578131] [Epoch 0] | Iter: 5200 | Optim: 5200 | Time: 1.91s | LR: 0.000399 | SkipOpt: 0 loss: 0.263523 | l2_loss: 0.019310 | psnr: 17.142223 | perceptual_loss: 0.488427 | norm_perceptual_loss: 25.294243 [04:02:19.607908] [Epoch 0] | Iter: 5250 | Optim: 5250 | Time: 0.77s | LR: 0.000399 | SkipOpt: 0 loss: 0.314335 | l2_loss: 0.020265 | psnr: 16.932520 | perceptual_loss: 0.588140 | norm_perceptual_loss: 29.022371 [04:03:20.440837] [Epoch 0] | Iter: 5300 | Optim: 5300 | Time: 1.08s | LR: 0.000399 | SkipOpt: 0 loss: 0.268908 | l2_loss: 0.017948 | psnr: 17.459848 | perceptual_loss: 0.501920 | norm_perceptual_loss: 27.965296 [04:04:20.282527] [Epoch 0] | Iter: 5350 | Optim: 5350 | Time: 1.90s | LR: 0.000399 | SkipOpt: 0 loss: 0.257040 | l2_loss: 0.021839 | psnr: 16.607647 | perceptual_loss: 0.470401 | norm_perceptual_loss: 21.539383 [04:05:24.453726] [Epoch 0] | Iter: 5400 | Optim: 5400 | Time: 1.09s | LR: 0.000399 | SkipOpt: 0 loss: 0.317193 | l2_loss: 0.022871 | psnr: 16.407063 | perceptual_loss: 0.588643 | norm_perceptual_loss: 25.737005 [04:06:25.071723] [Epoch 0] | Iter: 5450 | Optim: 5450 | Time: 0.75s | LR: 0.000398 | SkipOpt: 0 loss: 0.218568 | l2_loss: 0.014728 | psnr: 18.318554 | perceptual_loss: 0.407679 | norm_perceptual_loss: 27.680485 [04:07:24.344337] [Epoch 0] | Iter: 5500 | Optim: 5500 | Time: 1.90s | LR: 0.000398 | SkipOpt: 0 loss: 0.249871 | l2_loss: 0.018085 | psnr: 17.426899 | perceptual_loss: 0.463573 | norm_perceptual_loss: 25.633493 [04:08:25.118251] [Epoch 0] | Iter: 5550 | Optim: 5550 | Time: 0.75s | LR: 0.000398 | SkipOpt: 0 loss: 0.267982 | l2_loss: 0.018383 | psnr: 17.355898 | perceptual_loss: 0.499199 | norm_perceptual_loss: 27.155884 [04:09:25.939287] [Epoch 0] | Iter: 5600 | Optim: 5600 | Time: 1.27s | LR: 0.000398 | SkipOpt: 0 loss: 0.253182 | l2_loss: 0.018599 | psnr: 17.305105 | perceptual_loss: 0.469166 | norm_perceptual_loss: 25.225340 [04:10:25.172292] [Epoch 0] | Iter: 5650 | Optim: 5650 | Time: 1.91s | LR: 0.000398 | SkipOpt: 0 loss: 0.255757 | l2_loss: 0.020185 | psnr: 16.949654 | perceptual_loss: 0.471144 | norm_perceptual_loss: 23.340982 [04:11:27.070318] [Epoch 0] | Iter: 5700 | Optim: 5700 | Time: 1.90s | LR: 0.000398 | SkipOpt: 0 loss: 0.291750 | l2_loss: 0.025199 | psnr: 15.986108 | perceptual_loss: 0.533101 | norm_perceptual_loss: 21.155338 [04:12:35.477854] [Epoch 0] | Iter: 5750 | Optim: 5750 | Time: 1.07s | LR: 0.000398 | SkipOpt: 0 loss: 0.263200 | l2_loss: 0.018854 | psnr: 17.245956 | perceptual_loss: 0.488692 | norm_perceptual_loss: 25.919744 [04:13:33.904443] [Epoch 0] | Iter: 5800 | Optim: 5800 | Time: 1.90s | LR: 0.000398 | SkipOpt: 0 loss: 0.261593 | l2_loss: 0.020691 | psnr: 16.842157 | perceptual_loss: 0.481804 | norm_perceptual_loss: 23.285530 [04:14:36.266473] [Epoch 0] | Iter: 5850 | Optim: 5850 | Time: 1.08s | LR: 0.000398 | SkipOpt: 0 loss: 0.269581 | l2_loss: 0.021443 | psnr: 16.687090 | perceptual_loss: 0.496276 | norm_perceptual_loss: 23.143660 [04:15:43.641517] [Epoch 0] | Iter: 5900 | Optim: 5900 | Time: 1.90s | LR: 0.000398 | SkipOpt: 0 loss: 0.265245 | l2_loss: 0.018524 | psnr: 17.322769 | perceptual_loss: 0.493443 | norm_perceptual_loss: 26.638758 [04:16:48.255848] [Epoch 0] | Iter: 5950 | Optim: 5950 | Time: 1.07s | LR: 0.000398 | SkipOpt: 0 loss: 0.210855 | l2_loss: 0.011711 | psnr: 19.313923 | perceptual_loss: 0.398287 | norm_perceptual_loss: 34.008610 [04:17:56.302904] [Epoch 0] | Iter: 6000 | Optim: 6000 | Time: 1.27s | LR: 0.000398 | SkipOpt: 0 loss: 0.279799 | l2_loss: 0.022072 | psnr: 16.561630 | perceptual_loss: 0.515454 | norm_perceptual_loss: 23.353563 [04:18:04.091667] Saved checkpoint at iteration 6000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [04:18:04.092982] Saved latest checkpoint at iteration 6000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [04:19:02.831775] [Epoch 0] | Iter: 6050 | Optim: 6050 | Time: 1.91s | LR: 0.000398 | SkipOpt: 0 loss: 0.278016 | l2_loss: 0.021272 | psnr: 16.721958 | perceptual_loss: 0.513488 | norm_perceptual_loss: 24.139370 [04:20:06.058492] [Epoch 0] | Iter: 6100 | Optim: 6100 | Time: 1.90s | LR: 0.000398 | SkipOpt: 0 loss: 0.262351 | l2_loss: 0.019909 | psnr: 17.009546 | perceptual_loss: 0.484884 | norm_perceptual_loss: 24.355221 [04:21:09.509053] [Epoch 0] | Iter: 6150 | Optim: 6150 | Time: 0.85s | LR: 0.000397 | SkipOpt: 0 loss: 0.296252 | l2_loss: 0.025267 | psnr: 15.974483 | perceptual_loss: 0.541971 | norm_perceptual_loss: 21.449860 [04:22:13.484038] [Epoch 0] | Iter: 6200 | Optim: 6200 | Time: 1.08s | LR: 0.000397 | SkipOpt: 0 loss: 0.266037 | l2_loss: 0.020984 | psnr: 16.781118 | perceptual_loss: 0.490106 | norm_perceptual_loss: 23.356195 [04:23:15.267994] [Epoch 0] | Iter: 6250 | Optim: 6250 | Time: 0.75s | LR: 0.000397 | SkipOpt: 0 loss: 0.238698 | l2_loss: 0.014965 | psnr: 18.249136 | perceptual_loss: 0.447466 | norm_perceptual_loss: 29.900175 [04:24:23.462375] [Epoch 0] | Iter: 6300 | Optim: 6300 | Time: 1.06s | LR: 0.000397 | SkipOpt: 0 loss: 0.295342 | l2_loss: 0.024643 | psnr: 16.083059 | perceptual_loss: 0.541398 | norm_perceptual_loss: 21.969604 [04:25:26.962451] [Epoch 0] | Iter: 6350 | Optim: 6350 | Time: 1.08s | LR: 0.000397 | SkipOpt: 0 loss: 0.236008 | l2_loss: 0.016689 | psnr: 17.775621 | perceptual_loss: 0.438637 | norm_perceptual_loss: 26.282558 [04:26:30.824099] [Epoch 0] | Iter: 6400 | Optim: 6400 | Time: 1.93s | LR: 0.000397 | SkipOpt: 0 loss: 0.254769 | l2_loss: 0.018624 | psnr: 17.299278 | perceptual_loss: 0.472291 | norm_perceptual_loss: 25.359285 [04:27:35.859070] [Epoch 0] | Iter: 6450 | Optim: 6450 | Time: 1.93s | LR: 0.000397 | SkipOpt: 0 loss: 0.252368 | l2_loss: 0.015589 | psnr: 18.071896 | perceptual_loss: 0.473558 | norm_perceptual_loss: 30.378273 [04:28:39.802477] [Epoch 0] | Iter: 6500 | Optim: 6500 | Time: 1.07s | LR: 0.000397 | SkipOpt: 0 loss: 0.272232 | l2_loss: 0.022718 | psnr: 16.436337 | perceptual_loss: 0.499028 | norm_perceptual_loss: 21.966373 [04:29:45.364249] [Epoch 0] | Iter: 6550 | Optim: 6550 | Time: 1.90s | LR: 0.000397 | SkipOpt: 0 loss: 0.249890 | l2_loss: 0.019091 | psnr: 17.191683 | perceptual_loss: 0.461598 | norm_perceptual_loss: 24.178646 [04:30:50.528511] [Epoch 0] | Iter: 6600 | Optim: 6600 | Time: 1.93s | LR: 0.000397 | SkipOpt: 0 loss: 0.274000 | l2_loss: 0.021241 | psnr: 16.728260 | perceptual_loss: 0.505517 | norm_perceptual_loss: 23.799177 [04:32:04.556984] [Epoch 0] | Iter: 6650 | Optim: 6650 | Time: 0.86s | LR: 0.000397 | SkipOpt: 0 loss: 0.269090 | l2_loss: 0.014306 | psnr: 18.444864 | perceptual_loss: 0.509569 | norm_perceptual_loss: 35.619648 [04:33:11.027449] [Epoch 0] | Iter: 6700 | Optim: 6700 | Time: 1.21s | LR: 0.000396 | SkipOpt: 0 loss: 0.247321 | l2_loss: 0.017511 | psnr: 17.566988 | perceptual_loss: 0.459620 | norm_perceptual_loss: 26.248083 [04:34:24.029593] [Epoch 0] | Iter: 6750 | Optim: 6750 | Time: 0.98s | LR: 0.000396 | SkipOpt: 0 loss: 0.241632 | l2_loss: 0.015039 | psnr: 18.227875 | perceptual_loss: 0.453186 | norm_perceptual_loss: 30.134462 [04:35:25.749874] [Epoch 0] | Iter: 6800 | Optim: 6800 | Time: 1.00s | LR: 0.000396 | SkipOpt: 0 loss: 0.284874 | l2_loss: 0.020296 | psnr: 16.925982 | perceptual_loss: 0.529158 | norm_perceptual_loss: 26.072523 [04:36:35.809709] [Epoch 0] | Iter: 6850 | Optim: 6850 | Time: 1.92s | LR: 0.000396 | SkipOpt: 0 loss: 0.245214 | l2_loss: 0.017350 | psnr: 17.607073 | perceptual_loss: 0.455729 | norm_perceptual_loss: 26.267225 [04:37:38.253387] [Epoch 0] | Iter: 6900 | Optim: 6900 | Time: 1.05s | LR: 0.000396 | SkipOpt: 0 loss: 0.224781 | l2_loss: 0.016939 | psnr: 17.711159 | perceptual_loss: 0.415684 | norm_perceptual_loss: 24.540277 [04:38:50.299370] [Epoch 0] | Iter: 6950 | Optim: 6950 | Time: 1.92s | LR: 0.000396 | SkipOpt: 0 loss: 0.271346 | l2_loss: 0.022342 | psnr: 16.508810 | perceptual_loss: 0.498008 | norm_perceptual_loss: 22.290365 [04:39:58.328785] [Epoch 0] | Iter: 7000 | Optim: 7000 | Time: 1.37s | LR: 0.000396 | SkipOpt: 0 loss: 0.230804 | l2_loss: 0.014643 | psnr: 18.343750 | perceptual_loss: 0.432323 | norm_perceptual_loss: 29.524553 [04:40:09.785977] Saved checkpoint at iteration 7000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [04:40:09.786403] Saved latest checkpoint at iteration 7000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [04:40:28.393090] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [04:41:22.964711] [Epoch 0] | Iter: 7050 | Optim: 7050 | Time: 1.92s | LR: 0.000396 | SkipOpt: 0 loss: 0.271066 | l2_loss: 0.026446 | psnr: 15.776357 | perceptual_loss: 0.489240 | norm_perceptual_loss: 18.499388 [04:42:32.697552] [Epoch 0] | Iter: 7100 | Optim: 7100 | Time: 1.24s | LR: 0.000396 | SkipOpt: 0 loss: 0.219271 | l2_loss: 0.017395 | psnr: 17.595802 | perceptual_loss: 0.403753 | norm_perceptual_loss: 23.211132 [04:43:45.973939] [Epoch 0] | Iter: 7150 | Optim: 7150 | Time: 1.07s | LR: 0.000396 | SkipOpt: 0 loss: 0.268362 | l2_loss: 0.016500 | psnr: 17.825077 | perceptual_loss: 0.503724 | norm_perceptual_loss: 30.528109 [04:44:49.660696] [Epoch 0] | Iter: 7200 | Optim: 7200 | Time: 0.75s | LR: 0.000395 | SkipOpt: 0 loss: 0.236883 | l2_loss: 0.014036 | psnr: 18.527618 | perceptual_loss: 0.445695 | norm_perceptual_loss: 31.754059 [04:45:49.021311] [Epoch 0] | Iter: 7250 | Optim: 7250 | Time: 0.75s | LR: 0.000395 | SkipOpt: 0 loss: 0.247850 | l2_loss: 0.016148 | psnr: 17.918941 | perceptual_loss: 0.463404 | norm_perceptual_loss: 28.698166 [04:46:49.962753] [Epoch 0] | Iter: 7300 | Optim: 7300 | Time: 1.07s | LR: 0.000395 | SkipOpt: 0 loss: 0.204124 | l2_loss: 0.011801 | psnr: 19.280787 | perceptual_loss: 0.384646 | norm_perceptual_loss: 32.594151 [04:47:45.628813] [Epoch 0] | Iter: 7350 | Optim: 7350 | Time: 1.90s | LR: 0.000395 | SkipOpt: 0 loss: 0.270786 | l2_loss: 0.019704 | psnr: 17.054420 | perceptual_loss: 0.502164 | norm_perceptual_loss: 25.485157 [04:48:42.081535] [Epoch 0] | Iter: 7400 | Optim: 7400 | Time: 1.07s | LR: 0.000395 | SkipOpt: 0 loss: 0.247437 | l2_loss: 0.018319 | psnr: 17.370916 | perceptual_loss: 0.458236 | norm_perceptual_loss: 25.013834 [04:49:41.178379] [Epoch 0] | Iter: 7450 | Optim: 7450 | Time: 1.07s | LR: 0.000395 | SkipOpt: 0 loss: 0.243430 | l2_loss: 0.017586 | psnr: 17.548357 | perceptual_loss: 0.451688 | norm_perceptual_loss: 25.684708 [04:50:47.916484] [Epoch 0] | Iter: 7500 | Optim: 7500 | Time: 0.75s | LR: 0.000395 | SkipOpt: 0 loss: 0.231466 | l2_loss: 0.011882 | psnr: 19.251205 | perceptual_loss: 0.439169 | norm_perceptual_loss: 36.961662 [04:51:49.543741] [Epoch 0] | Iter: 7550 | Optim: 7550 | Time: 0.75s | LR: 0.000395 | SkipOpt: 0 loss: 0.269454 | l2_loss: 0.016207 | psnr: 17.903069 | perceptual_loss: 0.506495 | norm_perceptual_loss: 31.252287 [04:52:56.113949] [Epoch 0] | Iter: 7600 | Optim: 7600 | Time: 1.27s | LR: 0.000395 | SkipOpt: 0 loss: 0.289675 | l2_loss: 0.022255 | psnr: 16.525751 | perceptual_loss: 0.534839 | norm_perceptual_loss: 24.032480 [04:53:57.397570] [Epoch 0] | Iter: 7650 | Optim: 7650 | Time: 1.91s | LR: 0.000394 | SkipOpt: 0 loss: 0.245868 | l2_loss: 0.016883 | psnr: 17.725576 | perceptual_loss: 0.457971 | norm_perceptual_loss: 27.126604 [04:55:01.240732] [Epoch 0] | Iter: 7700 | Optim: 7700 | Time: 0.78s | LR: 0.000394 | SkipOpt: 0 loss: 0.188634 | l2_loss: 0.009925 | psnr: 20.032597 | perceptual_loss: 0.357418 | norm_perceptual_loss: 36.011036 [04:56:11.282479] [Epoch 0] | Iter: 7750 | Optim: 7750 | Time: 1.91s | LR: 0.000394 | SkipOpt: 0 loss: 0.265771 | l2_loss: 0.023099 | psnr: 16.364155 | perceptual_loss: 0.485345 | norm_perceptual_loss: 21.011934 [04:57:09.162014] [Epoch 0] | Iter: 7800 | Optim: 7800 | Time: 1.07s | LR: 0.000394 | SkipOpt: 0 loss: 0.258256 | l2_loss: 0.018195 | psnr: 17.400600 | perceptual_loss: 0.480124 | norm_perceptual_loss: 26.388412 [04:58:07.907023] [Epoch 0] | Iter: 7850 | Optim: 7850 | Time: 1.26s | LR: 0.000394 | SkipOpt: 0 loss: 0.251182 | l2_loss: 0.016167 | psnr: 17.913637 | perceptual_loss: 0.470030 | norm_perceptual_loss: 29.072964 [04:59:11.240202] [Epoch 0] | Iter: 7900 | Optim: 7900 | Time: 0.76s | LR: 0.000394 | SkipOpt: 0 loss: 0.277950 | l2_loss: 0.020234 | psnr: 16.939213 | perceptual_loss: 0.515432 | norm_perceptual_loss: 25.473749 [05:00:16.208035] [Epoch 0] | Iter: 7950 | Optim: 7950 | Time: 1.90s | LR: 0.000394 | SkipOpt: 0 loss: 0.253613 | l2_loss: 0.018169 | psnr: 17.406586 | perceptual_loss: 0.470888 | norm_perceptual_loss: 25.916473 [05:01:15.423977] [Epoch 0] | Iter: 8000 | Optim: 8000 | Time: 0.77s | LR: 0.000394 | SkipOpt: 0 loss: 0.220982 | l2_loss: 0.015673 | psnr: 18.048536 | perceptual_loss: 0.410619 | norm_perceptual_loss: 26.199484 [05:01:22.021763] Saved checkpoint at iteration 8000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [05:01:22.022380] Saved latest checkpoint at iteration 8000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [05:02:28.301485] [Epoch 0] | Iter: 8050 | Optim: 8050 | Time: 1.91s | LR: 0.000393 | SkipOpt: 0 loss: 0.257069 | l2_loss: 0.020581 | psnr: 16.865356 | perceptual_loss: 0.472975 | norm_perceptual_loss: 22.981268 [05:03:33.265804] [Epoch 0] | Iter: 8100 | Optim: 8100 | Time: 0.78s | LR: 0.000393 | SkipOpt: 0 loss: 0.271896 | l2_loss: 0.018941 | psnr: 17.226048 | perceptual_loss: 0.505911 | norm_perceptual_loss: 26.710279 [05:04:31.385795] [Epoch 0] | Iter: 8150 | Optim: 8150 | Time: 1.90s | LR: 0.000393 | SkipOpt: 0 loss: 0.263880 | l2_loss: 0.019055 | psnr: 17.199854 | perceptual_loss: 0.489649 | norm_perceptual_loss: 25.696291 [05:05:33.705082] [Epoch 0] | Iter: 8200 | Optim: 8200 | Time: 0.81s | LR: 0.000393 | SkipOpt: 0 loss: 0.225032 | l2_loss: 0.010456 | psnr: 19.806316 | perceptual_loss: 0.429153 | norm_perceptual_loss: 41.043423 [05:06:34.321751] [Epoch 0] | Iter: 8250 | Optim: 8250 | Time: 0.77s | LR: 0.000393 | SkipOpt: 0 loss: 0.263685 | l2_loss: 0.017710 | psnr: 17.517715 | perceptual_loss: 0.491950 | norm_perceptual_loss: 27.777439 [05:07:38.538751] [Epoch 0] | Iter: 8300 | Optim: 8300 | Time: 1.07s | LR: 0.000393 | SkipOpt: 0 loss: 0.224549 | l2_loss: 0.014189 | psnr: 18.480412 | perceptual_loss: 0.420720 | norm_perceptual_loss: 29.650682 [05:08:38.311255] [Epoch 0] | Iter: 8350 | Optim: 8350 | Time: 1.90s | LR: 0.000393 | SkipOpt: 0 loss: 0.240584 | l2_loss: 0.016331 | psnr: 17.869904 | perceptual_loss: 0.448505 | norm_perceptual_loss: 27.463625 [05:09:42.371596] [Epoch 0] | Iter: 8400 | Optim: 8400 | Time: 0.75s | LR: 0.000393 | SkipOpt: 0 loss: 0.228980 | l2_loss: 0.009787 | psnr: 20.093641 | perceptual_loss: 0.438386 | norm_perceptual_loss: 44.794090 [05:10:38.633309] [Epoch 0] | Iter: 8450 | Optim: 8450 | Time: 1.07s | LR: 0.000392 | SkipOpt: 0 loss: 0.259880 | l2_loss: 0.020139 | psnr: 16.959568 | perceptual_loss: 0.479481 | norm_perceptual_loss: 23.808310 [05:11:39.486526] [Epoch 0] | Iter: 8500 | Optim: 8500 | Time: 1.90s | LR: 0.000392 | SkipOpt: 0 loss: 0.252534 | l2_loss: 0.017927 | psnr: 17.464941 | perceptual_loss: 0.469214 | norm_perceptual_loss: 26.173710 [05:12:42.622786] [Epoch 0] | Iter: 8550 | Optim: 8550 | Time: 1.91s | LR: 0.000392 | SkipOpt: 0 loss: 0.243480 | l2_loss: 0.017620 | psnr: 17.539921 | perceptual_loss: 0.451719 | norm_perceptual_loss: 25.636606 [05:13:41.978787] [Epoch 0] | Iter: 8600 | Optim: 8600 | Time: 1.07s | LR: 0.000392 | SkipOpt: 0 loss: 0.251201 | l2_loss: 0.016779 | psnr: 17.752432 | perceptual_loss: 0.468846 | norm_perceptual_loss: 27.943005 [05:14:41.150320] [Epoch 0] | Iter: 8650 | Optim: 8650 | Time: 1.90s | LR: 0.000392 | SkipOpt: 0 loss: 0.293157 | l2_loss: 0.028148 | psnr: 15.505505 | perceptual_loss: 0.530017 | norm_perceptual_loss: 18.829554 [05:15:43.104977] [Epoch 0] | Iter: 8700 | Optim: 8700 | Time: 0.75s | LR: 0.000392 | SkipOpt: 0 loss: 0.218390 | l2_loss: 0.017568 | psnr: 17.552826 | perceptual_loss: 0.401644 | norm_perceptual_loss: 22.862488 [05:16:42.891917] [Epoch 0] | Iter: 8750 | Optim: 8750 | Time: 1.07s | LR: 0.000392 | SkipOpt: 0 loss: 0.291885 | l2_loss: 0.021425 | psnr: 16.690714 | perceptual_loss: 0.540918 | norm_perceptual_loss: 25.246601 [05:17:50.167435] [Epoch 0] | Iter: 8800 | Optim: 8800 | Time: 1.90s | LR: 0.000391 | SkipOpt: 0 loss: 0.259903 | l2_loss: 0.020491 | psnr: 16.884426 | perceptual_loss: 0.478824 | norm_perceptual_loss: 23.367857 [05:18:48.527246] [Epoch 0] | Iter: 8850 | Optim: 8850 | Time: 0.75s | LR: 0.000391 | SkipOpt: 0 loss: 0.266237 | l2_loss: 0.020996 | psnr: 16.778566 | perceptual_loss: 0.490481 | norm_perceptual_loss: 23.360332 [05:19:55.939735] [Epoch 0] | Iter: 8900 | Optim: 8900 | Time: 1.08s | LR: 0.000391 | SkipOpt: 0 loss: 0.256988 | l2_loss: 0.021296 | psnr: 16.716961 | perceptual_loss: 0.471384 | norm_perceptual_loss: 22.134581 [05:21:01.546198] [Epoch 0] | Iter: 8950 | Optim: 8950 | Time: 1.90s | LR: 0.000391 | SkipOpt: 0 loss: 0.263851 | l2_loss: 0.023103 | psnr: 16.363306 | perceptual_loss: 0.481495 | norm_perceptual_loss: 20.841204 [05:22:02.810565] [Epoch 0] | Iter: 9000 | Optim: 9000 | Time: 0.75s | LR: 0.000391 | SkipOpt: 0 loss: 0.197166 | l2_loss: 0.009496 | psnr: 20.224390 | perceptual_loss: 0.375339 | norm_perceptual_loss: 39.524113 [05:22:08.520014] Saved checkpoint at iteration 9000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [05:22:08.520447] Saved latest checkpoint at iteration 9000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [05:23:10.938275] [Epoch 0] | Iter: 9050 | Optim: 9050 | Time: 1.07s | LR: 0.000391 | SkipOpt: 0 loss: 0.260708 | l2_loss: 0.018776 | psnr: 17.263891 | perceptual_loss: 0.483864 | norm_perceptual_loss: 25.769890 [05:24:14.778316] [Epoch 0] | Iter: 9100 | Optim: 9100 | Time: 1.07s | LR: 0.000391 | SkipOpt: 0 loss: 0.247084 | l2_loss: 0.016813 | psnr: 17.743538 | perceptual_loss: 0.460542 | norm_perceptual_loss: 27.391933 [05:25:22.020202] [Epoch 0] | Iter: 9150 | Optim: 9150 | Time: 0.83s | LR: 0.000390 | SkipOpt: 0 loss: 0.287664 | l2_loss: 0.017262 | psnr: 17.629116 | perceptual_loss: 0.540804 | norm_perceptual_loss: 31.329330 [05:26:28.414101] [Epoch 0] | Iter: 9200 | Optim: 9200 | Time: 1.43s | LR: 0.000390 | SkipOpt: 0 loss: 0.265088 | l2_loss: 0.024627 | psnr: 16.085861 | perceptual_loss: 0.480922 | norm_perceptual_loss: 19.528143 [05:27:39.171304] [Epoch 0] | Iter: 9250 | Optim: 9250 | Time: 1.08s | LR: 0.000390 | SkipOpt: 0 loss: 0.258029 | l2_loss: 0.018816 | psnr: 17.254719 | perceptual_loss: 0.478425 | norm_perceptual_loss: 25.426447 [05:28:44.622782] [Epoch 0] | Iter: 9300 | Optim: 9300 | Time: 1.10s | LR: 0.000390 | SkipOpt: 0 loss: 0.243778 | l2_loss: 0.014370 | psnr: 18.425407 | perceptual_loss: 0.458815 | norm_perceptual_loss: 31.928509 [05:29:43.757566] [Epoch 0] | Iter: 9350 | Optim: 9350 | Time: 1.91s | LR: 0.000390 | SkipOpt: 0 loss: 0.255994 | l2_loss: 0.020987 | psnr: 16.780575 | perceptual_loss: 0.470014 | norm_perceptual_loss: 22.395905 [05:30:49.713955] [Epoch 0] | Iter: 9400 | Optim: 9400 | Time: 1.27s | LR: 0.000390 | SkipOpt: 0 loss: 0.247789 | l2_loss: 0.015010 | psnr: 18.236153 | perceptual_loss: 0.465557 | norm_perceptual_loss: 31.016161 [05:31:56.856902] [Epoch 0] | Iter: 9450 | Optim: 9450 | Time: 1.93s | LR: 0.000389 | SkipOpt: 0 loss: 0.248740 | l2_loss: 0.018043 | psnr: 17.436884 | perceptual_loss: 0.461395 | norm_perceptual_loss: 25.571783 [05:33:02.748648] [Epoch 0] | Iter: 9500 | Optim: 9500 | Time: 1.92s | LR: 0.000389 | SkipOpt: 0 loss: 0.289169 | l2_loss: 0.025559 | psnr: 15.924505 | perceptual_loss: 0.527219 | norm_perceptual_loss: 20.627239 [05:34:07.707479] [Epoch 0] | Iter: 9550 | Optim: 9550 | Time: 0.81s | LR: 0.000389 | SkipOpt: 0 loss: 0.224780 | l2_loss: 0.014283 | psnr: 18.451715 | perceptual_loss: 0.420994 | norm_perceptual_loss: 29.474550 [05:35:17.083226] [Epoch 0] | Iter: 9600 | Optim: 9600 | Time: 5.58s | LR: 0.000389 | SkipOpt: 0 loss: 0.222795 | l2_loss: 0.016746 | psnr: 17.760860 | perceptual_loss: 0.412099 | norm_perceptual_loss: 24.608604 [05:36:23.822815] [Epoch 0] | Iter: 9650 | Optim: 9650 | Time: 1.91s | LR: 0.000389 | SkipOpt: 0 loss: 0.274178 | l2_loss: 0.022607 | psnr: 16.457487 | perceptual_loss: 0.503141 | norm_perceptual_loss: 22.255552 [05:37:28.412784] [Epoch 0] | Iter: 9700 | Optim: 9700 | Time: 0.77s | LR: 0.000389 | SkipOpt: 0 loss: 0.289120 | l2_loss: 0.021439 | psnr: 16.688055 | perceptual_loss: 0.535364 | norm_perceptual_loss: 24.972061 [05:38:28.762560] [Epoch 0] | Iter: 9750 | Optim: 9750 | Time: 1.10s | LR: 0.000388 | SkipOpt: 0 loss: 0.255072 | l2_loss: 0.014924 | psnr: 18.261099 | perceptual_loss: 0.480296 | norm_perceptual_loss: 32.182423 [05:39:25.106385] [Epoch 0] | Iter: 9800 | Optim: 9800 | Time: 1.92s | LR: 0.000388 | SkipOpt: 0 loss: 0.260685 | l2_loss: 0.019437 | psnr: 17.113607 | perceptual_loss: 0.482495 | norm_perceptual_loss: 24.822966 [05:40:27.450116] [Epoch 0] | Iter: 9850 | Optim: 9850 | Time: 1.01s | LR: 0.000388 | SkipOpt: 0 loss: 0.228017 | l2_loss: 0.021128 | psnr: 16.751505 | perceptual_loss: 0.413779 | norm_perceptual_loss: 19.584774 [05:41:30.247358] [Epoch 0] | Iter: 9900 | Optim: 9900 | Time: 1.92s | LR: 0.000388 | SkipOpt: 0 loss: 0.255108 | l2_loss: 0.019460 | psnr: 17.108589 | perceptual_loss: 0.471297 | norm_perceptual_loss: 24.218822 [05:42:38.541640] [Epoch 0] | Iter: 9950 | Optim: 9950 | Time: 0.76s | LR: 0.000388 | SkipOpt: 0 loss: 0.269314 | l2_loss: 0.014644 | psnr: 18.343470 | perceptual_loss: 0.509340 | norm_perceptual_loss: 34.781998 [05:43:47.389317] [Epoch 0] | Iter: 10000 | Optim: 10000 | Time: 0.89s | LR: 0.000388 | SkipOpt: 0 loss: 0.219038 | l2_loss: 0.015780 | psnr: 18.018864 | perceptual_loss: 0.406516 | norm_perceptual_loss: 25.761084 [05:43:49.381919] Found 140 sequences [05:44:07.925791] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [05:44:09.817761] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [05:45:25.829317] Gathering scores from all processes... [05:45:25.830664] Padding local tensor of length 140 to max count: 140 [05:45:25.831332] Padding local tensor of length 140 to max count: 140 [05:45:25.831950] Padding local tensor of length 140 to max count: 140 [05:45:25.832392] Gathering scene names from all processes... [05:45:25.832675] Padding local tensor of length 140 on rank 0 to max count: 140 [05:45:25.834474] Evaluation metrics: [05:45:25.834661] psnr: 16.4890 [05:45:25.834744] ssim: 0.4400 [05:45:25.834826] lpips: 0.6055 [05:45:25.838908] Found 140 sequences [05:45:50.179847] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [05:45:51.425508] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [05:46:47.235703] Gathering scores from all processes... [05:46:47.236646] Padding local tensor of length 140 to max count: 140 [05:46:47.237316] Padding local tensor of length 140 to max count: 140 [05:46:47.237929] Padding local tensor of length 140 to max count: 140 [05:46:47.238281] Gathering scene names from all processes... [05:46:47.238392] Padding local tensor of length 140 on rank 0 to max count: 140 [05:46:47.239413] Evaluation metrics: [05:46:47.239582] psnr: 18.2984 [05:46:47.239667] ssim: 0.4804 [05:46:47.239834] lpips: 0.5571 [05:46:47.244007] Found 140 sequences [05:47:17.297942] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [05:47:18.382854] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [05:48:15.179649] Gathering scores from all processes... [05:48:15.180479] Padding local tensor of length 140 to max count: 140 [05:48:15.181106] Padding local tensor of length 140 to max count: 140 [05:48:15.181612] Padding local tensor of length 140 to max count: 140 [05:48:15.181870] Gathering scene names from all processes... [05:48:15.181989] Padding local tensor of length 140 on rank 0 to max count: 140 [05:48:15.182987] Evaluation metrics: [05:48:15.183152] psnr: 19.0743 [05:48:15.183234] ssim: 0.4995 [05:48:15.183308] lpips: 0.5384 [05:48:15.184810] [Eval][iter 10000] dl3dv_2view | psnr: 16.489031 | ssim: 0.439963 | lpips: 0.605489 || dl3dv_4view | psnr: 18.298374 | ssim: 0.480400 | lpips: 0.557058 || dl3dv_6view | psnr: 19.074331 | ssim: 0.499484 | lpips: 0.538382 [05:48:15.187708] [Eval][iter 10000] dl3dv_2view | psnr: 16.489031 | ssim: 0.439963 | lpips: 0.605489 || dl3dv_4view | psnr: 18.298374 | ssim: 0.480400 | lpips: 0.557058 || dl3dv_6view | psnr: 19.074331 | ssim: 0.499484 | lpips: 0.538382 [05:48:19.900983] Saved checkpoint at iteration 10000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [05:48:19.901538] Saved latest checkpoint at iteration 10000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [05:49:20.736918] [Epoch 0] | Iter: 10050 | Optim: 10050 | Time: 1.08s | LR: 0.000387 | SkipOpt: 0 loss: 0.223662 | l2_loss: 0.014413 | psnr: 18.412569 | perceptual_loss: 0.418499 | norm_perceptual_loss: 29.037008 [05:50:22.922502] [Epoch 0] | Iter: 10100 | Optim: 10100 | Time: 0.75s | LR: 0.000387 | SkipOpt: 0 loss: 0.204895 | l2_loss: 0.010603 | psnr: 19.745907 | perceptual_loss: 0.388586 | norm_perceptual_loss: 36.650318 [05:51:27.360626] [Epoch 0] | Iter: 10150 | Optim: 10150 | Time: 1.08s | LR: 0.000387 | SkipOpt: 0 loss: 0.216617 | l2_loss: 0.011401 | psnr: 19.430758 | perceptual_loss: 0.410433 | norm_perceptual_loss: 36.001282 [05:52:29.117764] [Epoch 0] | Iter: 10200 | Optim: 10200 | Time: 0.75s | LR: 0.000387 | SkipOpt: 0 loss: 0.224870 | l2_loss: 0.014483 | psnr: 18.391451 | perceptual_loss: 0.420774 | norm_perceptual_loss: 29.053198 [05:53:32.317486] [Epoch 0] | Iter: 10250 | Optim: 10250 | Time: 1.27s | LR: 0.000387 | SkipOpt: 0 loss: 0.237575 | l2_loss: 0.014357 | psnr: 18.429396 | perceptual_loss: 0.446437 | norm_perceptual_loss: 31.095648 [05:54:37.997623] [Epoch 0] | Iter: 10300 | Optim: 10300 | Time: 1.07s | LR: 0.000386 | SkipOpt: 0 loss: 0.231234 | l2_loss: 0.013598 | psnr: 18.665331 | perceptual_loss: 0.435273 | norm_perceptual_loss: 32.010677 [05:55:39.798749] [Epoch 0] | Iter: 10350 | Optim: 10350 | Time: 0.75s | LR: 0.000386 | SkipOpt: 0 loss: 0.288853 | l2_loss: 0.020111 | psnr: 16.965652 | perceptual_loss: 0.537484 | norm_perceptual_loss: 26.725801 [05:56:47.980252] [Epoch 0] | Iter: 10400 | Optim: 10400 | Time: 0.75s | LR: 0.000386 | SkipOpt: 0 loss: 0.183004 | l2_loss: 0.007467 | psnr: 21.268332 | perceptual_loss: 0.351073 | norm_perceptual_loss: 47.014389 [05:57:05.391631] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [05:57:46.874106] [Epoch 0] | Iter: 10450 | Optim: 10450 | Time: 1.27s | LR: 0.000386 | SkipOpt: 0 loss: 0.197940 | l2_loss: 0.009917 | psnr: 20.036392 | perceptual_loss: 0.376048 | norm_perceptual_loss: 37.921200 [05:58:45.986690] [Epoch 0] | Iter: 10500 | Optim: 10500 | Time: 1.08s | LR: 0.000386 | SkipOpt: 0 loss: 0.232866 | l2_loss: 0.016545 | psnr: 17.813347 | perceptual_loss: 0.432643 | norm_perceptual_loss: 26.149555 [05:59:46.179199] [Epoch 0] | Iter: 10550 | Optim: 10550 | Time: 0.75s | LR: 0.000386 | SkipOpt: 0 loss: 0.233777 | l2_loss: 0.016806 | psnr: 17.745346 | perceptual_loss: 0.433941 | norm_perceptual_loss: 25.820564 [06:00:49.836155] [Epoch 0] | Iter: 10600 | Optim: 10600 | Time: 1.90s | LR: 0.000385 | SkipOpt: 0 loss: 0.288183 | l2_loss: 0.023780 | psnr: 16.237909 | perceptual_loss: 0.528807 | norm_perceptual_loss: 22.237612 [06:01:53.661387] [Epoch 0] | Iter: 10650 | Optim: 10650 | Time: 1.90s | LR: 0.000385 | SkipOpt: 0 loss: 0.240544 | l2_loss: 0.017683 | psnr: 17.524549 | perceptual_loss: 0.445723 | norm_perceptual_loss: 25.206911 [06:03:01.001376] [Epoch 0] | Iter: 10700 | Optim: 10700 | Time: 1.07s | LR: 0.000385 | SkipOpt: 0 loss: 0.254658 | l2_loss: 0.019742 | psnr: 17.046021 | perceptual_loss: 0.469831 | norm_perceptual_loss: 23.798197 [06:04:06.163884] [Epoch 0] | Iter: 10750 | Optim: 10750 | Time: 0.75s | LR: 0.000385 | SkipOpt: 0 loss: 0.240489 | l2_loss: 0.012015 | psnr: 19.202621 | perceptual_loss: 0.456946 | norm_perceptual_loss: 38.030094 [06:05:07.151353] [Epoch 0] | Iter: 10800 | Optim: 10800 | Time: 0.85s | LR: 0.000385 | SkipOpt: 0 loss: 0.235727 | l2_loss: 0.015244 | psnr: 18.168905 | perceptual_loss: 0.440966 | norm_perceptual_loss: 28.926477 [06:06:06.322521] [Epoch 0] | Iter: 10850 | Optim: 10850 | Time: 1.07s | LR: 0.000384 | SkipOpt: 0 loss: 0.246338 | l2_loss: 0.020672 | psnr: 16.846239 | perceptual_loss: 0.451334 | norm_perceptual_loss: 21.833408 [06:07:11.598860] [Epoch 0] | Iter: 10900 | Optim: 10900 | Time: 1.90s | LR: 0.000384 | SkipOpt: 0 loss: 0.258819 | l2_loss: 0.022669 | psnr: 16.445677 | perceptual_loss: 0.472299 | norm_perceptual_loss: 20.834587 [06:08:17.685875] [Epoch 0] | Iter: 10950 | Optim: 10950 | Time: 1.27s | LR: 0.000384 | SkipOpt: 0 loss: 0.262480 | l2_loss: 0.018185 | psnr: 17.402790 | perceptual_loss: 0.488589 | norm_perceptual_loss: 26.867247 [06:09:19.110767] [Epoch 0] | Iter: 11000 | Optim: 11000 | Time: 1.07s | LR: 0.000384 | SkipOpt: 0 loss: 0.215283 | l2_loss: 0.013920 | psnr: 18.563564 | perceptual_loss: 0.402725 | norm_perceptual_loss: 28.931131 [06:09:26.583245] Saved checkpoint at iteration 11000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [06:09:26.583886] Saved latest checkpoint at iteration 11000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [06:10:28.970581] [Epoch 0] | Iter: 11050 | Optim: 11050 | Time: 0.89s | LR: 0.000384 | SkipOpt: 0 loss: 0.238148 | l2_loss: 0.014171 | psnr: 18.485926 | perceptual_loss: 0.447953 | norm_perceptual_loss: 31.610041 [06:11:29.228533] [Epoch 0] | Iter: 11100 | Optim: 11100 | Time: 1.91s | LR: 0.000383 | SkipOpt: 0 loss: 0.238758 | l2_loss: 0.017295 | psnr: 17.620913 | perceptual_loss: 0.442927 | norm_perceptual_loss: 25.610806 [06:12:29.017375] [Epoch 0] | Iter: 11150 | Optim: 11150 | Time: 0.75s | LR: 0.000383 | SkipOpt: 0 loss: 0.253033 | l2_loss: 0.020892 | psnr: 16.800175 | perceptual_loss: 0.464282 | norm_perceptual_loss: 22.222811 [06:13:34.013570] [Epoch 0] | Iter: 11200 | Optim: 11200 | Time: 1.90s | LR: 0.000383 | SkipOpt: 0 loss: 0.261038 | l2_loss: 0.021022 | psnr: 16.773281 | perceptual_loss: 0.480031 | norm_perceptual_loss: 22.834829 [06:14:32.284285] [Epoch 0] | Iter: 11250 | Optim: 11250 | Time: 1.90s | LR: 0.000383 | SkipOpt: 0 loss: 0.258510 | l2_loss: 0.018224 | psnr: 17.393526 | perceptual_loss: 0.480572 | norm_perceptual_loss: 26.370062 [06:15:29.914651] [Epoch 0] | Iter: 11300 | Optim: 11300 | Time: 0.75s | LR: 0.000383 | SkipOpt: 0 loss: 0.217893 | l2_loss: 0.013484 | psnr: 18.701818 | perceptual_loss: 0.408819 | norm_perceptual_loss: 30.318830 [06:16:34.145934] [Epoch 0] | Iter: 11350 | Optim: 11350 | Time: 1.08s | LR: 0.000382 | SkipOpt: 0 loss: 0.284690 | l2_loss: 0.019726 | psnr: 17.049671 | perceptual_loss: 0.529928 | norm_perceptual_loss: 26.864809 [06:17:37.872994] [Epoch 0] | Iter: 11400 | Optim: 11400 | Time: 1.07s | LR: 0.000382 | SkipOpt: 0 loss: 0.261854 | l2_loss: 0.021514 | psnr: 16.672810 | perceptual_loss: 0.480681 | norm_perceptual_loss: 22.342798 [06:18:38.879291] [Epoch 0] | Iter: 11450 | Optim: 11450 | Time: 0.89s | LR: 0.000382 | SkipOpt: 0 loss: 0.199143 | l2_loss: 0.014947 | psnr: 18.254593 | perceptual_loss: 0.368392 | norm_perceptual_loss: 24.647331 [06:19:50.207906] [Epoch 0] | Iter: 11500 | Optim: 11500 | Time: 1.10s | LR: 0.000382 | SkipOpt: 0 loss: 0.215705 | l2_loss: 0.017284 | psnr: 17.623667 | perceptual_loss: 0.396844 | norm_perceptual_loss: 22.960743 [06:20:50.816241] [Epoch 0] | Iter: 11550 | Optim: 11550 | Time: 1.07s | LR: 0.000382 | SkipOpt: 0 loss: 0.261729 | l2_loss: 0.017895 | psnr: 17.472609 | perceptual_loss: 0.487668 | norm_perceptual_loss: 27.251171 [06:21:52.364661] [Epoch 0] | Iter: 11600 | Optim: 11600 | Time: 0.76s | LR: 0.000381 | SkipOpt: 0 loss: 0.190654 | l2_loss: 0.009143 | psnr: 20.388893 | perceptual_loss: 0.363021 | norm_perceptual_loss: 39.702812 [06:22:56.636077] [Epoch 0] | Iter: 11650 | Optim: 11650 | Time: 1.07s | LR: 0.000381 | SkipOpt: 0 loss: 0.213208 | l2_loss: 0.012037 | psnr: 19.194820 | perceptual_loss: 0.402342 | norm_perceptual_loss: 33.425453 [06:24:00.277394] [Epoch 0] | Iter: 11700 | Optim: 11700 | Time: 0.85s | LR: 0.000381 | SkipOpt: 0 loss: 0.199056 | l2_loss: 0.014155 | psnr: 18.490864 | perceptual_loss: 0.369802 | norm_perceptual_loss: 26.124973 [06:25:02.755257] [Epoch 0] | Iter: 11750 | Optim: 11750 | Time: 1.08s | LR: 0.000381 | SkipOpt: 0 loss: 0.219310 | l2_loss: 0.011595 | psnr: 19.357349 | perceptual_loss: 0.415431 | norm_perceptual_loss: 35.828941 [06:26:10.090414] [Epoch 0] | Iter: 11800 | Optim: 11800 | Time: 1.91s | LR: 0.000380 | SkipOpt: 0 loss: 0.305332 | l2_loss: 0.024412 | psnr: 16.123880 | perceptual_loss: 0.561840 | norm_perceptual_loss: 23.014450 [06:27:06.806152] [Epoch 0] | Iter: 11850 | Optim: 11850 | Time: 0.76s | LR: 0.000380 | SkipOpt: 0 loss: 0.246886 | l2_loss: 0.015636 | psnr: 18.058716 | perceptual_loss: 0.462500 | norm_perceptual_loss: 29.578974 [06:28:09.089932] [Epoch 0] | Iter: 11900 | Optim: 11900 | Time: 1.90s | LR: 0.000380 | SkipOpt: 0 loss: 0.225137 | l2_loss: 0.018956 | psnr: 17.222462 | perceptual_loss: 0.412361 | norm_perceptual_loss: 21.753237 [06:29:09.664349] [Epoch 0] | Iter: 11950 | Optim: 11950 | Time: 0.75s | LR: 0.000380 | SkipOpt: 0 loss: 0.221729 | l2_loss: 0.012004 | psnr: 19.206825 | perceptual_loss: 0.419451 | norm_perceptual_loss: 34.943279 [06:30:11.843118] [Epoch 0] | Iter: 12000 | Optim: 12000 | Time: 0.75s | LR: 0.000380 | SkipOpt: 0 loss: 0.223266 | l2_loss: 0.012894 | psnr: 18.896078 | perceptual_loss: 0.420743 | norm_perceptual_loss: 32.630589 [06:30:18.164127] Saved checkpoint at iteration 12000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [06:30:18.164744] Saved latest checkpoint at iteration 12000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [06:31:12.265350] [Epoch 0] | Iter: 12050 | Optim: 12050 | Time: 0.75s | LR: 0.000379 | SkipOpt: 0 loss: 0.201482 | l2_loss: 0.011860 | psnr: 19.259323 | perceptual_loss: 0.379246 | norm_perceptual_loss: 31.978123 [06:32:23.660487] [Epoch 0] | Iter: 12100 | Optim: 12100 | Time: 1.91s | LR: 0.000379 | SkipOpt: 0 loss: 0.256909 | l2_loss: 0.021219 | psnr: 16.732712 | perceptual_loss: 0.471379 | norm_perceptual_loss: 22.214741 [06:33:32.262677] [Epoch 0] | Iter: 12150 | Optim: 12150 | Time: 1.90s | LR: 0.000379 | SkipOpt: 0 loss: 0.251687 | l2_loss: 0.020861 | psnr: 16.806683 | perceptual_loss: 0.461653 | norm_perceptual_loss: 22.130123 [06:34:37.463784] [Epoch 0] | Iter: 12200 | Optim: 12200 | Time: 1.91s | LR: 0.000379 | SkipOpt: 0 loss: 0.248630 | l2_loss: 0.020715 | psnr: 16.837105 | perceptual_loss: 0.455829 | norm_perceptual_loss: 22.004536 [06:35:38.665486] [Epoch 0] | Iter: 12250 | Optim: 12250 | Time: 1.07s | LR: 0.000378 | SkipOpt: 0 loss: 0.226561 | l2_loss: 0.018042 | psnr: 17.437267 | perceptual_loss: 0.417039 | norm_perceptual_loss: 23.115528 [06:36:39.666979] [Epoch 0] | Iter: 12300 | Optim: 12300 | Time: 0.75s | LR: 0.000378 | SkipOpt: 0 loss: 0.250388 | l2_loss: 0.016279 | psnr: 17.883675 | perceptual_loss: 0.468218 | norm_perceptual_loss: 28.761744 [06:37:42.258894] [Epoch 0] | Iter: 12350 | Optim: 12350 | Time: 1.08s | LR: 0.000378 | SkipOpt: 0 loss: 0.193453 | l2_loss: 0.010108 | psnr: 19.953547 | perceptual_loss: 0.366690 | norm_perceptual_loss: 36.278877 [06:38:45.723158] [Epoch 0] | Iter: 12400 | Optim: 12400 | Time: 1.90s | LR: 0.000378 | SkipOpt: 0 loss: 0.218244 | l2_loss: 0.020473 | psnr: 16.888229 | perceptual_loss: 0.395543 | norm_perceptual_loss: 19.320414 [06:39:45.962793] [Epoch 0] | Iter: 12450 | Optim: 12450 | Time: 0.75s | LR: 0.000378 | SkipOpt: 0 loss: 0.237900 | l2_loss: 0.017752 | psnr: 17.507505 | perceptual_loss: 0.440295 | norm_perceptual_loss: 24.802441 [06:40:51.492539] [Epoch 0] | Iter: 12500 | Optim: 12500 | Time: 1.27s | LR: 0.000377 | SkipOpt: 0 loss: 0.225677 | l2_loss: 0.013291 | psnr: 18.764444 | perceptual_loss: 0.424772 | norm_perceptual_loss: 31.959501 [06:41:54.790977] [Epoch 0] | Iter: 12550 | Optim: 12550 | Time: 1.07s | LR: 0.000377 | SkipOpt: 0 loss: 0.259682 | l2_loss: 0.017308 | psnr: 17.617657 | perceptual_loss: 0.484749 | norm_perceptual_loss: 28.008055 [06:43:00.184092] [Epoch 0] | Iter: 12600 | Optim: 12600 | Time: 1.08s | LR: 0.000377 | SkipOpt: 0 loss: 0.249470 | l2_loss: 0.017074 | psnr: 17.676628 | perceptual_loss: 0.464792 | norm_perceptual_loss: 27.222086 [06:43:56.388061] [Epoch 0] | Iter: 12650 | Optim: 12650 | Time: 0.86s | LR: 0.000377 | SkipOpt: 0 loss: 0.203418 | l2_loss: 0.011373 | psnr: 19.441231 | perceptual_loss: 0.384091 | norm_perceptual_loss: 33.772011 [06:44:59.881554] [Epoch 0] | Iter: 12700 | Optim: 12700 | Time: 1.12s | LR: 0.000376 | SkipOpt: 0 loss: 0.229818 | l2_loss: 0.014822 | psnr: 18.290873 | perceptual_loss: 0.429991 | norm_perceptual_loss: 29.009924 [06:46:04.722426] [Epoch 0] | Iter: 12750 | Optim: 12750 | Time: 1.90s | LR: 0.000376 | SkipOpt: 0 loss: 0.274502 | l2_loss: 0.024264 | psnr: 16.150454 | perceptual_loss: 0.500476 | norm_perceptual_loss: 20.626644 [06:47:01.854110] [Epoch 0] | Iter: 12800 | Optim: 12800 | Time: 0.75s | LR: 0.000376 | SkipOpt: 0 loss: 0.259248 | l2_loss: 0.015322 | psnr: 18.146729 | perceptual_loss: 0.487850 | norm_perceptual_loss: 31.839001 [06:48:01.052320] [Epoch 0] | Iter: 12850 | Optim: 12850 | Time: 1.91s | LR: 0.000376 | SkipOpt: 0 loss: 0.228324 | l2_loss: 0.016793 | psnr: 17.748718 | perceptual_loss: 0.423062 | norm_perceptual_loss: 25.192789 [06:49:02.966388] [Epoch 0] | Iter: 12900 | Optim: 12900 | Time: 0.82s | LR: 0.000375 | SkipOpt: 0 loss: 0.230176 | l2_loss: 0.014212 | psnr: 18.473579 | perceptual_loss: 0.431928 | norm_perceptual_loss: 30.392715 [06:49:58.326090] [Epoch 0] | Iter: 12950 | Optim: 12950 | Time: 0.75s | LR: 0.000375 | SkipOpt: 0 loss: 0.187687 | l2_loss: 0.009056 | psnr: 20.430714 | perceptual_loss: 0.357263 | norm_perceptual_loss: 39.451061 [06:50:59.073290] [Epoch 0] | Iter: 13000 | Optim: 13000 | Time: 0.76s | LR: 0.000375 | SkipOpt: 0 loss: 0.218973 | l2_loss: 0.011847 | psnr: 19.264072 | perceptual_loss: 0.414253 | norm_perceptual_loss: 34.968143 [06:51:05.796162] Saved checkpoint at iteration 13000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [06:51:05.796622] Saved latest checkpoint at iteration 13000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [06:52:11.376505] [Epoch 0] | Iter: 13050 | Optim: 13050 | Time: 1.91s | LR: 0.000375 | SkipOpt: 0 loss: 0.247388 | l2_loss: 0.021002 | psnr: 16.777308 | perceptual_loss: 0.452772 | norm_perceptual_loss: 21.558064 [06:53:11.647914] [Epoch 0] | Iter: 13100 | Optim: 13100 | Time: 1.90s | LR: 0.000374 | SkipOpt: 0 loss: 0.248381 | l2_loss: 0.021352 | psnr: 16.705629 | perceptual_loss: 0.454059 | norm_perceptual_loss: 21.265472 [06:53:47.919512] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [06:54:13.302021] [Epoch 0] | Iter: 13150 | Optim: 13150 | Time: 0.76s | LR: 0.000374 | SkipOpt: 0 loss: 0.237929 | l2_loss: 0.011147 | psnr: 19.528236 | perceptual_loss: 0.453563 | norm_perceptual_loss: 40.687508 [06:55:18.168229] [Epoch 0] | Iter: 13200 | Optim: 13200 | Time: 1.91s | LR: 0.000374 | SkipOpt: 0 loss: 0.259925 | l2_loss: 0.021111 | psnr: 16.754992 | perceptual_loss: 0.477628 | norm_perceptual_loss: 22.625029 [06:56:24.402955] [Epoch 0] | Iter: 13250 | Optim: 13250 | Time: 0.75s | LR: 0.000374 | SkipOpt: 0 loss: 0.199175 | l2_loss: 0.008839 | psnr: 20.535988 | perceptual_loss: 0.380672 | norm_perceptual_loss: 43.067509 [06:57:23.895633] [Epoch 0] | Iter: 13300 | Optim: 13300 | Time: 0.75s | LR: 0.000373 | SkipOpt: 0 loss: 0.224838 | l2_loss: 0.013234 | psnr: 18.782951 | perceptual_loss: 0.423207 | norm_perceptual_loss: 31.977755 [06:58:16.175885] [Epoch 0] | Iter: 13350 | Optim: 13350 | Time: 0.75s | LR: 0.000373 | SkipOpt: 0 loss: 0.206361 | l2_loss: 0.010943 | psnr: 19.608671 | perceptual_loss: 0.390837 | norm_perceptual_loss: 35.715950 [06:59:20.304897] [Epoch 0] | Iter: 13400 | Optim: 13400 | Time: 1.90s | LR: 0.000373 | SkipOpt: 0 loss: 0.254350 | l2_loss: 0.021716 | psnr: 16.632189 | perceptual_loss: 0.465267 | norm_perceptual_loss: 21.425028 [07:00:24.159425] [Epoch 0] | Iter: 13450 | Optim: 13450 | Time: 1.07s | LR: 0.000373 | SkipOpt: 0 loss: 0.208877 | l2_loss: 0.016029 | psnr: 17.951054 | perceptual_loss: 0.385697 | norm_perceptual_loss: 24.063099 [07:01:28.605005] [Epoch 0] | Iter: 13500 | Optim: 13500 | Time: 1.08s | LR: 0.000372 | SkipOpt: 0 loss: 0.279371 | l2_loss: 0.025071 | psnr: 16.008347 | perceptual_loss: 0.508601 | norm_perceptual_loss: 20.286703 [07:02:33.493073] [Epoch 0] | Iter: 13550 | Optim: 13550 | Time: 1.07s | LR: 0.000372 | SkipOpt: 0 loss: 0.199933 | l2_loss: 0.016399 | psnr: 17.851805 | perceptual_loss: 0.367068 | norm_perceptual_loss: 22.383446 [07:03:38.335182] [Epoch 0] | Iter: 13600 | Optim: 13600 | Time: 1.07s | LR: 0.000372 | SkipOpt: 0 loss: 0.181866 | l2_loss: 0.010096 | psnr: 19.958414 | perceptual_loss: 0.343539 | norm_perceptual_loss: 34.026505 [07:04:33.348237] [Epoch 0] | Iter: 13650 | Optim: 13650 | Time: 0.75s | LR: 0.000372 | SkipOpt: 0 loss: 0.223233 | l2_loss: 0.016354 | psnr: 17.863678 | perceptual_loss: 0.413758 | norm_perceptual_loss: 25.299637 [07:05:37.603154] [Epoch 0] | Iter: 13700 | Optim: 13700 | Time: 1.07s | LR: 0.000371 | SkipOpt: 0 loss: 0.265518 | l2_loss: 0.017688 | psnr: 17.523170 | perceptual_loss: 0.495659 | norm_perceptual_loss: 28.022060 [07:06:34.216713] [Epoch 0] | Iter: 13750 | Optim: 13750 | Time: 0.76s | LR: 0.000371 | SkipOpt: 0 loss: 0.236294 | l2_loss: 0.014037 | psnr: 18.527325 | perceptual_loss: 0.444515 | norm_perceptual_loss: 31.667891 [07:07:38.564874] [Epoch 0] | Iter: 13800 | Optim: 13800 | Time: 0.75s | LR: 0.000371 | SkipOpt: 0 loss: 0.210448 | l2_loss: 0.011243 | psnr: 19.491144 | perceptual_loss: 0.398410 | norm_perceptual_loss: 35.436039 [07:08:42.452159] [Epoch 0] | Iter: 13850 | Optim: 13850 | Time: 0.85s | LR: 0.000371 | SkipOpt: 0 loss: 0.192553 | l2_loss: 0.011813 | psnr: 19.276272 | perceptual_loss: 0.361480 | norm_perceptual_loss: 30.599279 [07:09:42.975904] [Epoch 0] | Iter: 13900 | Optim: 13900 | Time: 0.75s | LR: 0.000370 | SkipOpt: 0 loss: 0.204274 | l2_loss: 0.010939 | psnr: 19.610146 | perceptual_loss: 0.386669 | norm_perceptual_loss: 35.347130 [07:10:48.025303] [Epoch 0] | Iter: 13950 | Optim: 13950 | Time: 1.08s | LR: 0.000370 | SkipOpt: 0 loss: 0.247171 | l2_loss: 0.016451 | psnr: 17.838099 | perceptual_loss: 0.461440 | norm_perceptual_loss: 28.049503 [07:11:44.519878] [Epoch 0] | Iter: 14000 | Optim: 14000 | Time: 1.90s | LR: 0.000370 | SkipOpt: 0 loss: 0.253664 | l2_loss: 0.018004 | psnr: 17.446379 | perceptual_loss: 0.471320 | norm_perceptual_loss: 26.179054 [07:11:50.794643] Saved checkpoint at iteration 14000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [07:11:50.795116] Saved latest checkpoint at iteration 14000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [07:12:51.232058] [Epoch 0] | Iter: 14050 | Optim: 14050 | Time: 0.75s | LR: 0.000369 | SkipOpt: 0 loss: 0.178019 | l2_loss: 0.009006 | psnr: 20.454678 | perceptual_loss: 0.338025 | norm_perceptual_loss: 37.533325 [07:13:57.233099] [Epoch 0] | Iter: 14100 | Optim: 14100 | Time: 1.08s | LR: 0.000369 | SkipOpt: 0 loss: 0.233969 | l2_loss: 0.016317 | psnr: 17.873699 | perceptual_loss: 0.435304 | norm_perceptual_loss: 26.678549 [07:14:58.972265] [Epoch 0] | Iter: 14150 | Optim: 14150 | Time: 0.76s | LR: 0.000369 | SkipOpt: 0 loss: 0.187599 | l2_loss: 0.008034 | psnr: 20.950773 | perceptual_loss: 0.359131 | norm_perceptual_loss: 44.702339 [07:15:58.232985] [Epoch 0] | Iter: 14200 | Optim: 14200 | Time: 0.76s | LR: 0.000369 | SkipOpt: 0 loss: 0.237679 | l2_loss: 0.014481 | psnr: 18.391911 | perceptual_loss: 0.446395 | norm_perceptual_loss: 30.825481 [07:17:00.728598] [Epoch 0] | Iter: 14250 | Optim: 14250 | Time: 1.08s | LR: 0.000368 | SkipOpt: 0 loss: 0.207697 | l2_loss: 0.012391 | psnr: 19.068945 | perceptual_loss: 0.390612 | norm_perceptual_loss: 31.523930 [07:18:04.127055] [Epoch 0] | Iter: 14300 | Optim: 14300 | Time: 1.08s | LR: 0.000368 | SkipOpt: 0 loss: 0.222861 | l2_loss: 0.011727 | psnr: 19.308296 | perceptual_loss: 0.422270 | norm_perceptual_loss: 36.009708 [07:19:08.899138] [Epoch 0] | Iter: 14350 | Optim: 14350 | Time: 0.75s | LR: 0.000368 | SkipOpt: 0 loss: 0.260551 | l2_loss: 0.016560 | psnr: 17.809433 | perceptual_loss: 0.487983 | norm_perceptual_loss: 29.467821 [07:20:16.253224] [Epoch 0] | Iter: 14400 | Optim: 14400 | Time: 1.07s | LR: 0.000368 | SkipOpt: 0 loss: 0.184837 | l2_loss: 0.012767 | psnr: 18.939060 | perceptual_loss: 0.344139 | norm_perceptual_loss: 26.955076 [07:21:21.129711] [Epoch 0] | Iter: 14450 | Optim: 14450 | Time: 1.08s | LR: 0.000367 | SkipOpt: 0 loss: 0.226766 | l2_loss: 0.015748 | psnr: 18.027775 | perceptual_loss: 0.422036 | norm_perceptual_loss: 26.799503 [07:22:28.040090] [Epoch 0] | Iter: 14500 | Optim: 14500 | Time: 0.75s | LR: 0.000367 | SkipOpt: 0 loss: 0.218827 | l2_loss: 0.016137 | psnr: 17.921869 | perceptual_loss: 0.405381 | norm_perceptual_loss: 25.121754 [07:23:31.997404] [Epoch 0] | Iter: 14550 | Optim: 14550 | Time: 1.91s | LR: 0.000367 | SkipOpt: 0 loss: 0.261989 | l2_loss: 0.021808 | psnr: 16.613792 | perceptual_loss: 0.480361 | norm_perceptual_loss: 22.026567 [07:24:39.452483] [Epoch 0] | Iter: 14600 | Optim: 14600 | Time: 1.08s | LR: 0.000366 | SkipOpt: 0 loss: 0.242505 | l2_loss: 0.015148 | psnr: 18.196575 | perceptual_loss: 0.454714 | norm_perceptual_loss: 30.018980 [07:25:43.961988] [Epoch 0] | Iter: 14650 | Optim: 14650 | Time: 1.90s | LR: 0.000366 | SkipOpt: 0 loss: 0.259171 | l2_loss: 0.022242 | psnr: 16.528322 | perceptual_loss: 0.473858 | norm_perceptual_loss: 21.304947 [07:26:44.916420] [Epoch 0] | Iter: 14700 | Optim: 14700 | Time: 0.75s | LR: 0.000366 | SkipOpt: 0 loss: 0.227810 | l2_loss: 0.012539 | psnr: 19.017317 | perceptual_loss: 0.430542 | norm_perceptual_loss: 34.335835 [07:27:43.741405] [Epoch 0] | Iter: 14750 | Optim: 14750 | Time: 0.85s | LR: 0.000366 | SkipOpt: 0 loss: 0.255016 | l2_loss: 0.019849 | psnr: 17.022694 | perceptual_loss: 0.470334 | norm_perceptual_loss: 23.696039 [07:28:47.017281] [Epoch 0] | Iter: 14800 | Optim: 14800 | Time: 0.75s | LR: 0.000365 | SkipOpt: 0 loss: 0.234477 | l2_loss: 0.015067 | psnr: 18.219736 | perceptual_loss: 0.438819 | norm_perceptual_loss: 29.124554 [07:29:52.746439] [Epoch 0] | Iter: 14850 | Optim: 14850 | Time: 1.90s | LR: 0.000365 | SkipOpt: 0 loss: 0.245230 | l2_loss: 0.020015 | psnr: 16.986374 | perceptual_loss: 0.450430 | norm_perceptual_loss: 22.504276 [07:30:48.033196] [Epoch 0] | Iter: 14900 | Optim: 14900 | Time: 1.91s | LR: 0.000365 | SkipOpt: 0 loss: 0.239519 | l2_loss: 0.018517 | psnr: 17.324379 | perceptual_loss: 0.442005 | norm_perceptual_loss: 23.870672 [07:31:55.965572] [Epoch 0] | Iter: 14950 | Optim: 14950 | Time: 1.91s | LR: 0.000364 | SkipOpt: 0 loss: 0.262318 | l2_loss: 0.020532 | psnr: 16.875721 | perceptual_loss: 0.483573 | norm_perceptual_loss: 23.552347 [07:32:58.938938] [Epoch 0] | Iter: 15000 | Optim: 15000 | Time: 1.90s | LR: 0.000364 | SkipOpt: 0 loss: 0.278497 | l2_loss: 0.022447 | psnr: 16.488409 | perceptual_loss: 0.512099 | norm_perceptual_loss: 22.813675 [07:33:00.804511] Found 140 sequences [07:33:25.291529] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [07:33:26.164875] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [07:34:17.727172] Gathering scores from all processes... [07:34:17.728578] Padding local tensor of length 140 to max count: 140 [07:34:17.729346] Padding local tensor of length 140 to max count: 140 [07:34:17.729884] Padding local tensor of length 140 to max count: 140 [07:34:17.730221] Gathering scene names from all processes... [07:34:17.730342] Padding local tensor of length 140 on rank 0 to max count: 140 [07:34:17.731518] Evaluation metrics: [07:34:17.731720] psnr: 16.5579 [07:34:17.731813] ssim: 0.4465 [07:34:17.731906] lpips: 0.5648 [07:34:17.735621] Found 140 sequences [07:34:42.784610] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [07:34:43.325658] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [07:35:35.946132] Gathering scores from all processes... [07:35:35.947303] Padding local tensor of length 140 to max count: 140 [07:35:35.947978] Padding local tensor of length 140 to max count: 140 [07:35:35.948480] Padding local tensor of length 140 to max count: 140 [07:35:35.948746] Gathering scene names from all processes... [07:35:35.948852] Padding local tensor of length 140 on rank 0 to max count: 140 [07:35:35.950054] Evaluation metrics: [07:35:35.950226] psnr: 18.5575 [07:35:35.950306] ssim: 0.4976 [07:35:35.950381] lpips: 0.4987 [07:35:35.954506] Found 140 sequences [07:36:00.413819] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [07:36:01.325468] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [07:36:56.992372] Gathering scores from all processes... [07:36:56.993716] Padding local tensor of length 140 to max count: 140 [07:36:56.994484] Padding local tensor of length 140 to max count: 140 [07:36:56.995003] Padding local tensor of length 140 to max count: 140 [07:36:56.995271] Gathering scene names from all processes... [07:36:56.995383] Padding local tensor of length 140 on rank 0 to max count: 140 [07:36:56.996492] Evaluation metrics: [07:36:56.996662] psnr: 19.4092 [07:36:56.996744] ssim: 0.5217 [07:36:56.996820] lpips: 0.4732 [07:36:56.998621] [Eval][iter 15000] dl3dv_2view | psnr: 16.557905 | ssim: 0.446517 | lpips: 0.564823 || dl3dv_4view | psnr: 18.557508 | ssim: 0.497642 | lpips: 0.498675 || dl3dv_6view | psnr: 19.409191 | ssim: 0.521651 | lpips: 0.473214 [07:36:57.002331] [Eval][iter 15000] dl3dv_2view | psnr: 16.557905 | ssim: 0.446517 | lpips: 0.564823 || dl3dv_4view | psnr: 18.557508 | ssim: 0.497642 | lpips: 0.498675 || dl3dv_6view | psnr: 19.409191 | ssim: 0.521651 | lpips: 0.473214 [07:37:01.791992] Saved checkpoint at iteration 15000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [07:37:01.793213] Saved latest checkpoint at iteration 15000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [07:38:03.723113] [Epoch 0] | Iter: 15050 | Optim: 15050 | Time: 0.85s | LR: 0.000364 | SkipOpt: 0 loss: 0.232801 | l2_loss: 0.015726 | psnr: 18.033712 | perceptual_loss: 0.434149 | norm_perceptual_loss: 27.606421 [07:38:18.424166] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [07:39:04.624619] [Epoch 0] | Iter: 15100 | Optim: 15100 | Time: 0.86s | LR: 0.000364 | SkipOpt: 0 loss: 0.220022 | l2_loss: 0.013230 | psnr: 18.784397 | perceptual_loss: 0.413584 | norm_perceptual_loss: 31.261013 [07:40:04.279038] [Epoch 0] | Iter: 15150 | Optim: 15150 | Time: 1.07s | LR: 0.000363 | SkipOpt: 0 loss: 0.237949 | l2_loss: 0.015600 | psnr: 18.068716 | perceptual_loss: 0.444698 | norm_perceptual_loss: 28.506050 [07:41:00.548646] [Epoch 0] | Iter: 15200 | Optim: 15200 | Time: 1.07s | LR: 0.000363 | SkipOpt: 0 loss: 0.253715 | l2_loss: 0.015853 | psnr: 17.998821 | perceptual_loss: 0.475724 | norm_perceptual_loss: 30.008013 [07:42:01.858244] [Epoch 0] | Iter: 15250 | Optim: 15250 | Time: 1.08s | LR: 0.000363 | SkipOpt: 0 loss: 0.224486 | l2_loss: 0.013921 | psnr: 18.563263 | perceptual_loss: 0.421130 | norm_perceptual_loss: 30.251205 [07:43:04.179595] [Epoch 0] | Iter: 15300 | Optim: 15300 | Time: 1.07s | LR: 0.000362 | SkipOpt: 0 loss: 0.232124 | l2_loss: 0.016107 | psnr: 17.929962 | perceptual_loss: 0.432035 | norm_perceptual_loss: 26.823492 [07:44:05.994123] [Epoch 0] | Iter: 15350 | Optim: 15350 | Time: 0.75s | LR: 0.000362 | SkipOpt: 0 loss: 0.191330 | l2_loss: 0.011987 | psnr: 19.212940 | perceptual_loss: 0.358686 | norm_perceptual_loss: 29.923206 [07:44:52.263935] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [07:45:09.411588] [Epoch 0] | Iter: 15400 | Optim: 15400 | Time: 1.26s | LR: 0.000362 | SkipOpt: 0 loss: 0.232421 | l2_loss: 0.021521 | psnr: 16.671309 | perceptual_loss: 0.421799 | norm_perceptual_loss: 19.599102 [07:46:10.120587] [Epoch 0] | Iter: 15450 | Optim: 15450 | Time: 0.75s | LR: 0.000362 | SkipOpt: 0 loss: 0.231622 | l2_loss: 0.016021 | psnr: 17.953171 | perceptual_loss: 0.431202 | norm_perceptual_loss: 26.915207 [07:47:14.086540] [Epoch 0] | Iter: 15500 | Optim: 15500 | Time: 1.07s | LR: 0.000361 | SkipOpt: 0 loss: 0.229803 | l2_loss: 0.016578 | psnr: 17.804804 | perceptual_loss: 0.426452 | norm_perceptual_loss: 25.724703 [07:48:13.873840] [Epoch 0] | Iter: 15550 | Optim: 15550 | Time: 1.27s | LR: 0.000361 | SkipOpt: 0 loss: 0.224795 | l2_loss: 0.017630 | psnr: 17.537485 | perceptual_loss: 0.414330 | norm_perceptual_loss: 23.501486 [07:49:12.247341] [Epoch 0] | Iter: 15600 | Optim: 15600 | Time: 1.08s | LR: 0.000361 | SkipOpt: 0 loss: 0.223392 | l2_loss: 0.014788 | psnr: 18.300861 | perceptual_loss: 0.417208 | norm_perceptual_loss: 28.212301 [07:50:22.047627] [Epoch 0] | Iter: 15650 | Optim: 15650 | Time: 0.75s | LR: 0.000360 | SkipOpt: 0 loss: 0.227488 | l2_loss: 0.014612 | psnr: 18.352934 | perceptual_loss: 0.425753 | norm_perceptual_loss: 29.137419 [07:51:23.306250] [Epoch 0] | Iter: 15700 | Optim: 15700 | Time: 1.07s | LR: 0.000360 | SkipOpt: 0 loss: 0.213277 | l2_loss: 0.012046 | psnr: 19.191425 | perceptual_loss: 0.402462 | norm_perceptual_loss: 33.409298 [07:52:25.607926] [Epoch 0] | Iter: 15750 | Optim: 15750 | Time: 1.07s | LR: 0.000360 | SkipOpt: 0 loss: 0.213539 | l2_loss: 0.012356 | psnr: 19.081383 | perceptual_loss: 0.402366 | norm_perceptual_loss: 32.565624 [07:53:30.729443] [Epoch 0] | Iter: 15800 | Optim: 15800 | Time: 1.91s | LR: 0.000359 | SkipOpt: 0 loss: 0.209431 | l2_loss: 0.013051 | psnr: 18.843628 | perceptual_loss: 0.392761 | norm_perceptual_loss: 30.094769 [07:54:38.696949] [Epoch 0] | Iter: 15850 | Optim: 15850 | Time: 1.27s | LR: 0.000359 | SkipOpt: 0 loss: 0.224554 | l2_loss: 0.017450 | psnr: 17.582102 | perceptual_loss: 0.414208 | norm_perceptual_loss: 23.737171 [07:55:47.111568] [Epoch 0] | Iter: 15900 | Optim: 15900 | Time: 0.75s | LR: 0.000359 | SkipOpt: 0 loss: 0.190124 | l2_loss: 0.009695 | psnr: 20.134657 | perceptual_loss: 0.360858 | norm_perceptual_loss: 37.222221 [07:56:48.873104] [Epoch 0] | Iter: 15950 | Optim: 15950 | Time: 1.07s | LR: 0.000358 | SkipOpt: 0 loss: 0.191083 | l2_loss: 0.010506 | psnr: 19.785698 | perceptual_loss: 0.361155 | norm_perceptual_loss: 34.376625 [07:57:47.554928] [Epoch 0] | Iter: 16000 | Optim: 16000 | Time: 1.27s | LR: 0.000358 | SkipOpt: 0 loss: 0.231030 | l2_loss: 0.019066 | psnr: 17.197363 | perceptual_loss: 0.423929 | norm_perceptual_loss: 22.234581 [07:57:53.414914] Saved checkpoint at iteration 16000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [07:57:53.415475] Saved latest checkpoint at iteration 16000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [07:58:54.650096] [Epoch 0] | Iter: 16050 | Optim: 16050 | Time: 1.27s | LR: 0.000358 | SkipOpt: 0 loss: 0.254258 | l2_loss: 0.018793 | psnr: 17.259958 | perceptual_loss: 0.470930 | norm_perceptual_loss: 25.058311 [07:59:55.370052] [Epoch 0] | Iter: 16100 | Optim: 16100 | Time: 1.08s | LR: 0.000358 | SkipOpt: 0 loss: 0.212978 | l2_loss: 0.011702 | psnr: 19.317287 | perceptual_loss: 0.402551 | norm_perceptual_loss: 34.399261 [08:00:53.952516] [Epoch 0] | Iter: 16150 | Optim: 16150 | Time: 0.85s | LR: 0.000357 | SkipOpt: 0 loss: 0.209266 | l2_loss: 0.011560 | psnr: 19.370348 | perceptual_loss: 0.395413 | norm_perceptual_loss: 34.204651 [08:01:56.072783] [Epoch 0] | Iter: 16200 | Optim: 16200 | Time: 1.07s | LR: 0.000357 | SkipOpt: 0 loss: 0.260504 | l2_loss: 0.018160 | psnr: 17.408958 | perceptual_loss: 0.484690 | norm_perceptual_loss: 26.690689 [08:02:56.963599] [Epoch 0] | Iter: 16250 | Optim: 16250 | Time: 1.07s | LR: 0.000357 | SkipOpt: 0 loss: 0.234686 | l2_loss: 0.014177 | psnr: 18.484142 | perceptual_loss: 0.441018 | norm_perceptual_loss: 31.107887 [08:04:01.192281] [Epoch 0] | Iter: 16300 | Optim: 16300 | Time: 1.07s | LR: 0.000356 | SkipOpt: 0 loss: 0.235470 | l2_loss: 0.016100 | psnr: 17.931744 | perceptual_loss: 0.438740 | norm_perceptual_loss: 27.250925 [08:05:04.302994] [Epoch 0] | Iter: 16350 | Optim: 16350 | Time: 0.75s | LR: 0.000356 | SkipOpt: 0 loss: 0.224676 | l2_loss: 0.011846 | psnr: 19.264164 | perceptual_loss: 0.425660 | norm_perceptual_loss: 35.931839 [08:06:03.236331] [Epoch 0] | Iter: 16400 | Optim: 16400 | Time: 1.07s | LR: 0.000356 | SkipOpt: 0 loss: 0.219954 | l2_loss: 0.013378 | psnr: 18.736023 | perceptual_loss: 0.413151 | norm_perceptual_loss: 30.882374 [08:07:04.677488] [Epoch 0] | Iter: 16450 | Optim: 16450 | Time: 1.91s | LR: 0.000355 | SkipOpt: 0 loss: 0.206865 | l2_loss: 0.012630 | psnr: 18.986048 | perceptual_loss: 0.388471 | norm_perceptual_loss: 30.758348 [08:08:06.157143] [Epoch 0] | Iter: 16500 | Optim: 16500 | Time: 1.06s | LR: 0.000355 | SkipOpt: 0 loss: 0.188180 | l2_loss: 0.011347 | psnr: 19.451080 | perceptual_loss: 0.353666 | norm_perceptual_loss: 31.167442 [08:09:05.577555] [Epoch 0] | Iter: 16550 | Optim: 16550 | Time: 1.90s | LR: 0.000355 | SkipOpt: 0 loss: 0.238587 | l2_loss: 0.017952 | psnr: 17.458763 | perceptual_loss: 0.441269 | norm_perceptual_loss: 24.579868 [08:10:08.098302] [Epoch 0] | Iter: 16600 | Optim: 16600 | Time: 1.08s | LR: 0.000354 | SkipOpt: 0 loss: 0.201503 | l2_loss: 0.014178 | psnr: 18.483837 | perceptual_loss: 0.374650 | norm_perceptual_loss: 26.424633 [08:11:09.520197] [Epoch 0] | Iter: 16650 | Optim: 16650 | Time: 1.07s | LR: 0.000354 | SkipOpt: 0 loss: 0.222090 | l2_loss: 0.018563 | psnr: 17.313597 | perceptual_loss: 0.407054 | norm_perceptual_loss: 21.928665 [08:12:08.459878] [Epoch 0] | Iter: 16700 | Optim: 16700 | Time: 0.75s | LR: 0.000354 | SkipOpt: 0 loss: 0.236201 | l2_loss: 0.015028 | psnr: 18.231052 | perceptual_loss: 0.442346 | norm_perceptual_loss: 29.435238 [08:13:13.639659] [Epoch 0] | Iter: 16750 | Optim: 16750 | Time: 0.85s | LR: 0.000353 | SkipOpt: 0 loss: 0.237334 | l2_loss: 0.013969 | psnr: 18.548445 | perceptual_loss: 0.446732 | norm_perceptual_loss: 31.980925 [08:14:14.678678] [Epoch 0] | Iter: 16800 | Optim: 16800 | Time: 0.85s | LR: 0.000353 | SkipOpt: 0 loss: 0.238085 | l2_loss: 0.021269 | psnr: 16.722610 | perceptual_loss: 0.433633 | norm_perceptual_loss: 20.388432 [08:15:20.533203] [Epoch 0] | Iter: 16850 | Optim: 16850 | Time: 0.85s | LR: 0.000353 | SkipOpt: 0 loss: 0.187612 | l2_loss: 0.011082 | psnr: 19.554010 | perceptual_loss: 0.353061 | norm_perceptual_loss: 31.860336 [08:16:26.000378] [Epoch 0] | Iter: 16900 | Optim: 16900 | Time: 1.91s | LR: 0.000352 | SkipOpt: 0 loss: 0.237154 | l2_loss: 0.018331 | psnr: 17.368105 | perceptual_loss: 0.437647 | norm_perceptual_loss: 23.874477 [08:16:40.766492] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [08:17:21.388448] [Epoch 0] | Iter: 16950 | Optim: 16950 | Time: 0.75s | LR: 0.000352 | SkipOpt: 0 loss: 0.185117 | l2_loss: 0.009907 | psnr: 20.040695 | perceptual_loss: 0.350421 | norm_perceptual_loss: 35.371948 [08:18:23.808264] [Epoch 0] | Iter: 17000 | Optim: 17000 | Time: 1.07s | LR: 0.000352 | SkipOpt: 0 loss: 0.206503 | l2_loss: 0.012858 | psnr: 18.908157 | perceptual_loss: 0.387289 | norm_perceptual_loss: 30.119722 [08:18:29.904258] Saved checkpoint at iteration 17000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [08:18:29.904824] Saved latest checkpoint at iteration 17000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [08:19:31.636217] [Epoch 0] | Iter: 17050 | Optim: 17050 | Time: 1.91s | LR: 0.000351 | SkipOpt: 0 loss: 0.265403 | l2_loss: 0.019991 | psnr: 16.991722 | perceptual_loss: 0.490824 | norm_perceptual_loss: 24.552647 [08:20:38.865685] [Epoch 0] | Iter: 17100 | Optim: 17100 | Time: 1.07s | LR: 0.000351 | SkipOpt: 0 loss: 0.225565 | l2_loss: 0.014686 | psnr: 18.331013 | perceptual_loss: 0.421759 | norm_perceptual_loss: 28.718731 [08:21:40.536788] [Epoch 0] | Iter: 17150 | Optim: 17150 | Time: 1.91s | LR: 0.000351 | SkipOpt: 0 loss: 0.205965 | l2_loss: 0.015041 | psnr: 18.227375 | perceptual_loss: 0.381850 | norm_perceptual_loss: 25.388086 [08:22:41.473079] [Epoch 0] | Iter: 17200 | Optim: 17200 | Time: 1.90s | LR: 0.000350 | SkipOpt: 0 loss: 0.236949 | l2_loss: 0.018456 | psnr: 17.338737 | perceptual_loss: 0.436987 | norm_perceptual_loss: 23.677856 [08:23:45.869194] [Epoch 0] | Iter: 17250 | Optim: 17250 | Time: 1.07s | LR: 0.000350 | SkipOpt: 0 loss: 0.216332 | l2_loss: 0.017210 | psnr: 17.642157 | perceptual_loss: 0.398244 | norm_perceptual_loss: 23.140091 [08:24:47.544601] [Epoch 0] | Iter: 17300 | Optim: 17300 | Time: 1.91s | LR: 0.000350 | SkipOpt: 0 loss: 0.242268 | l2_loss: 0.016978 | psnr: 17.701248 | perceptual_loss: 0.450581 | norm_perceptual_loss: 26.539804 [08:25:50.025270] [Epoch 0] | Iter: 17350 | Optim: 17350 | Time: 1.90s | LR: 0.000349 | SkipOpt: 0 loss: 0.228849 | l2_loss: 0.016665 | psnr: 17.781994 | perceptual_loss: 0.424368 | norm_perceptual_loss: 25.464901 [08:26:52.779132] [Epoch 0] | Iter: 17400 | Optim: 17400 | Time: 0.75s | LR: 0.000349 | SkipOpt: 0 loss: 0.186162 | l2_loss: 0.010687 | psnr: 19.711540 | perceptual_loss: 0.350951 | norm_perceptual_loss: 32.839813 [08:27:52.442158] [Epoch 0] | Iter: 17450 | Optim: 17450 | Time: 0.75s | LR: 0.000349 | SkipOpt: 0 loss: 0.187442 | l2_loss: 0.008925 | psnr: 20.493814 | perceptual_loss: 0.357034 | norm_perceptual_loss: 40.002823 [08:28:50.817354] [Epoch 0] | Iter: 17500 | Optim: 17500 | Time: 0.75s | LR: 0.000348 | SkipOpt: 0 loss: 0.261164 | l2_loss: 0.021101 | psnr: 16.756899 | perceptual_loss: 0.480124 | norm_perceptual_loss: 22.753258 [08:29:14.732554] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [08:29:59.090977] [Epoch 0] | Iter: 17550 | Optim: 17550 | Time: 1.07s | LR: 0.000348 | SkipOpt: 0 loss: 0.229227 | l2_loss: 0.014473 | psnr: 18.394453 | perceptual_loss: 0.429508 | norm_perceptual_loss: 29.676762 [08:31:00.418579] [Epoch 0] | Iter: 17600 | Optim: 17600 | Time: 1.07s | LR: 0.000348 | SkipOpt: 0 loss: 0.234714 | l2_loss: 0.015519 | psnr: 18.091236 | perceptual_loss: 0.438389 | norm_perceptual_loss: 28.247715 [08:32:02.452202] [Epoch 0] | Iter: 17650 | Optim: 17650 | Time: 0.77s | LR: 0.000347 | SkipOpt: 0 loss: 0.193963 | l2_loss: 0.012757 | psnr: 18.942463 | perceptual_loss: 0.362412 | norm_perceptual_loss: 28.408569 [08:33:06.446695] [Epoch 0] | Iter: 17700 | Optim: 17700 | Time: 0.75s | LR: 0.000347 | SkipOpt: 0 loss: 0.231323 | l2_loss: 0.014241 | psnr: 18.464735 | perceptual_loss: 0.434165 | norm_perceptual_loss: 30.487965 [08:34:04.769448] [Epoch 0] | Iter: 17750 | Optim: 17750 | Time: 1.90s | LR: 0.000347 | SkipOpt: 0 loss: 0.246558 | l2_loss: 0.022980 | psnr: 16.386494 | perceptual_loss: 0.447155 | norm_perceptual_loss: 19.458426 [08:35:02.210648] [Epoch 0] | Iter: 17800 | Optim: 17800 | Time: 1.90s | LR: 0.000346 | SkipOpt: 0 loss: 0.268720 | l2_loss: 0.024009 | psnr: 16.196234 | perceptual_loss: 0.489422 | norm_perceptual_loss: 20.384804 [08:36:03.733135] [Epoch 0] | Iter: 17850 | Optim: 17850 | Time: 1.07s | LR: 0.000346 | SkipOpt: 0 loss: 0.201160 | l2_loss: 0.010601 | psnr: 19.746447 | perceptual_loss: 0.381118 | norm_perceptual_loss: 35.950375 [08:37:03.936733] [Epoch 0] | Iter: 17900 | Optim: 17900 | Time: 1.08s | LR: 0.000346 | SkipOpt: 0 loss: 0.250873 | l2_loss: 0.017533 | psnr: 17.561529 | perceptual_loss: 0.466680 | norm_perceptual_loss: 26.617825 [08:38:04.917782] [Epoch 0] | Iter: 17950 | Optim: 17950 | Time: 1.07s | LR: 0.000345 | SkipOpt: 0 loss: 0.203832 | l2_loss: 0.014108 | psnr: 18.505274 | perceptual_loss: 0.379448 | norm_perceptual_loss: 26.895500 [08:39:05.776596] [Epoch 0] | Iter: 18000 | Optim: 18000 | Time: 0.76s | LR: 0.000345 | SkipOpt: 0 loss: 0.199337 | l2_loss: 0.014317 | psnr: 18.441505 | perceptual_loss: 0.370040 | norm_perceptual_loss: 25.846340 [08:39:11.533162] Saved checkpoint at iteration 18000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [08:39:11.533814] Saved latest checkpoint at iteration 18000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [08:40:12.697290] [Epoch 0] | Iter: 18050 | Optim: 18050 | Time: 1.91s | LR: 0.000345 | SkipOpt: 0 loss: 0.213242 | l2_loss: 0.013270 | psnr: 18.771360 | perceptual_loss: 0.399944 | norm_perceptual_loss: 30.139425 [08:41:19.058635] [Epoch 0] | Iter: 18100 | Optim: 18100 | Time: 1.91s | LR: 0.000344 | SkipOpt: 0 loss: 0.198842 | l2_loss: 0.014823 | psnr: 18.290751 | perceptual_loss: 0.368038 | norm_perceptual_loss: 24.829494 [08:42:24.625579] [Epoch 0] | Iter: 18150 | Optim: 18150 | Time: 1.07s | LR: 0.000344 | SkipOpt: 0 loss: 0.256824 | l2_loss: 0.015562 | psnr: 18.079323 | perceptual_loss: 0.482524 | norm_perceptual_loss: 31.006401 [08:43:25.305341] [Epoch 0] | Iter: 18200 | Optim: 18200 | Time: 1.07s | LR: 0.000344 | SkipOpt: 0 loss: 0.177108 | l2_loss: 0.009739 | psnr: 20.114643 | perceptual_loss: 0.334736 | norm_perceptual_loss: 34.369003 [08:44:21.426013] [Epoch 0] | Iter: 18250 | Optim: 18250 | Time: 0.81s | LR: 0.000343 | SkipOpt: 0 loss: 0.179701 | l2_loss: 0.008090 | psnr: 20.920498 | perceptual_loss: 0.343222 | norm_perceptual_loss: 42.425323 [08:45:27.703226] [Epoch 0] | Iter: 18300 | Optim: 18300 | Time: 1.07s | LR: 0.000343 | SkipOpt: 0 loss: 0.223544 | l2_loss: 0.018433 | psnr: 17.343954 | perceptual_loss: 0.410221 | norm_perceptual_loss: 22.254240 [08:46:30.272252] [Epoch 0] | Iter: 18350 | Optim: 18350 | Time: 1.91s | LR: 0.000342 | SkipOpt: 0 loss: 0.241672 | l2_loss: 0.018383 | psnr: 17.355749 | perceptual_loss: 0.446576 | norm_perceptual_loss: 24.292402 [08:47:32.686877] [Epoch 0] | Iter: 18400 | Optim: 18400 | Time: 1.07s | LR: 0.000342 | SkipOpt: 0 loss: 0.182773 | l2_loss: 0.010508 | psnr: 19.784933 | perceptual_loss: 0.344531 | norm_perceptual_loss: 32.788548 [08:48:35.833613] [Epoch 0] | Iter: 18450 | Optim: 18450 | Time: 0.75s | LR: 0.000342 | SkipOpt: 0 loss: 0.192378 | l2_loss: 0.011562 | psnr: 19.369682 | perceptual_loss: 0.361632 | norm_perceptual_loss: 31.277702 [08:49:39.287046] [Epoch 0] | Iter: 18500 | Optim: 18500 | Time: 1.08s | LR: 0.000341 | SkipOpt: 0 loss: 0.217924 | l2_loss: 0.015037 | psnr: 18.228338 | perceptual_loss: 0.405774 | norm_perceptual_loss: 26.984690 [08:50:47.406495] [Epoch 0] | Iter: 18550 | Optim: 18550 | Time: 0.76s | LR: 0.000341 | SkipOpt: 0 loss: 0.196010 | l2_loss: 0.012421 | psnr: 19.058491 | perceptual_loss: 0.367179 | norm_perceptual_loss: 29.561529 [08:51:50.442083] [Epoch 0] | Iter: 18600 | Optim: 18600 | Time: 1.90s | LR: 0.000341 | SkipOpt: 0 loss: 0.247360 | l2_loss: 0.021406 | psnr: 16.694733 | perceptual_loss: 0.451909 | norm_perceptual_loss: 21.111732 [08:52:56.674351] [Epoch 0] | Iter: 18650 | Optim: 18650 | Time: 1.91s | LR: 0.000340 | SkipOpt: 0 loss: 0.246740 | l2_loss: 0.021325 | psnr: 16.711203 | perceptual_loss: 0.450831 | norm_perceptual_loss: 21.141407 [08:53:58.889242] [Epoch 0] | Iter: 18700 | Optim: 18700 | Time: 0.75s | LR: 0.000340 | SkipOpt: 0 loss: 0.199232 | l2_loss: 0.012280 | psnr: 19.108164 | perceptual_loss: 0.373906 | norm_perceptual_loss: 30.449360 [08:54:59.530208] [Epoch 0] | Iter: 18750 | Optim: 18750 | Time: 1.27s | LR: 0.000340 | SkipOpt: 0 loss: 0.289655 | l2_loss: 0.024902 | psnr: 16.037682 | perceptual_loss: 0.529506 | norm_perceptual_loss: 21.263691 [08:55:57.456767] [Epoch 0] | Iter: 18800 | Optim: 18800 | Time: 0.75s | LR: 0.000339 | SkipOpt: 0 loss: 0.234797 | l2_loss: 0.014708 | psnr: 18.324545 | perceptual_loss: 0.440180 | norm_perceptual_loss: 29.928461 [08:57:01.970405] [Epoch 0] | Iter: 18850 | Optim: 18850 | Time: 0.76s | LR: 0.000339 | SkipOpt: 0 loss: 0.191034 | l2_loss: 0.009084 | psnr: 20.417290 | perceptual_loss: 0.363900 | norm_perceptual_loss: 40.060066 [08:58:02.194631] [Epoch 0] | Iter: 18900 | Optim: 18900 | Time: 1.27s | LR: 0.000339 | SkipOpt: 0 loss: 0.240782 | l2_loss: 0.018656 | psnr: 17.291716 | perceptual_loss: 0.444251 | norm_perceptual_loss: 23.812222 [08:59:03.553100] [Epoch 0] | Iter: 18950 | Optim: 18950 | Time: 1.91s | LR: 0.000338 | SkipOpt: 0 loss: 0.212924 | l2_loss: 0.013901 | psnr: 18.569633 | perceptual_loss: 0.398047 | norm_perceptual_loss: 28.634998 [09:00:04.491654] [Epoch 0] | Iter: 19000 | Optim: 19000 | Time: 0.75s | LR: 0.000338 | SkipOpt: 0 loss: 0.213572 | l2_loss: 0.012571 | psnr: 19.006237 | perceptual_loss: 0.402002 | norm_perceptual_loss: 31.978075 [09:00:10.040340] Saved checkpoint at iteration 19000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [09:00:10.040757] Saved latest checkpoint at iteration 19000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [09:01:09.750283] [Epoch 0] | Iter: 19050 | Optim: 19050 | Time: 1.07s | LR: 0.000337 | SkipOpt: 0 loss: 0.214959 | l2_loss: 0.012434 | psnr: 19.053864 | perceptual_loss: 0.405049 | norm_perceptual_loss: 32.575752 [09:02:06.634112] [Epoch 0] | Iter: 19100 | Optim: 19100 | Time: 1.07s | LR: 0.000337 | SkipOpt: 0 loss: 0.226220 | l2_loss: 0.015049 | psnr: 18.225067 | perceptual_loss: 0.422343 | norm_perceptual_loss: 28.065474 [09:03:06.432563] [Epoch 0] | Iter: 19150 | Optim: 19150 | Time: 1.07s | LR: 0.000337 | SkipOpt: 0 loss: 0.205829 | l2_loss: 0.012711 | psnr: 18.958368 | perceptual_loss: 0.386237 | norm_perceptual_loss: 30.387196 [09:03:24.961971] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [09:04:07.039471] [Epoch 0] | Iter: 19200 | Optim: 19200 | Time: 0.85s | LR: 0.000336 | SkipOpt: 0 loss: 0.207175 | l2_loss: 0.011831 | psnr: 19.269928 | perceptual_loss: 0.390689 | norm_perceptual_loss: 33.023556 [09:05:12.244559] [Epoch 0] | Iter: 19250 | Optim: 19250 | Time: 0.85s | LR: 0.000336 | SkipOpt: 0 loss: 0.253447 | l2_loss: 0.021280 | psnr: 16.720196 | perceptual_loss: 0.464334 | norm_perceptual_loss: 21.819738 [09:06:13.263996] [Epoch 0] | Iter: 19300 | Optim: 19300 | Time: 0.76s | LR: 0.000336 | SkipOpt: 0 loss: 0.219177 | l2_loss: 0.013907 | psnr: 18.567640 | perceptual_loss: 0.410541 | norm_perceptual_loss: 29.520258 [09:07:10.615019] [Epoch 0] | Iter: 19350 | Optim: 19350 | Time: 1.07s | LR: 0.000335 | SkipOpt: 0 loss: 0.229323 | l2_loss: 0.015629 | psnr: 18.060606 | perceptual_loss: 0.427388 | norm_perceptual_loss: 27.345289 [09:08:15.729111] [Epoch 0] | Iter: 19400 | Optim: 19400 | Time: 0.75s | LR: 0.000335 | SkipOpt: 0 loss: 0.197983 | l2_loss: 0.012495 | psnr: 19.032475 | perceptual_loss: 0.370975 | norm_perceptual_loss: 29.688770 [09:09:18.336904] [Epoch 0] | Iter: 19450 | Optim: 19450 | Time: 1.91s | LR: 0.000334 | SkipOpt: 0 loss: 0.226149 | l2_loss: 0.017703 | psnr: 17.519459 | perceptual_loss: 0.416891 | norm_perceptual_loss: 23.548790 [09:10:21.855263] [Epoch 0] | Iter: 19500 | Optim: 19500 | Time: 1.91s | LR: 0.000334 | SkipOpt: 0 loss: 0.213693 | l2_loss: 0.015241 | psnr: 18.169754 | perceptual_loss: 0.396904 | norm_perceptual_loss: 26.041170 [09:11:18.739796] [Epoch 0] | Iter: 19550 | Optim: 19550 | Time: 1.91s | LR: 0.000334 | SkipOpt: 0 loss: 0.262244 | l2_loss: 0.023406 | psnr: 16.306782 | perceptual_loss: 0.477677 | norm_perceptual_loss: 20.408581 [09:12:23.296029] [Epoch 0] | Iter: 19600 | Optim: 19600 | Time: 0.75s | LR: 0.000333 | SkipOpt: 0 loss: 0.171980 | l2_loss: 0.007000 | psnr: 21.549316 | perceptual_loss: 0.329961 | norm_perceptual_loss: 47.140549 [09:13:26.307159] [Epoch 0] | Iter: 19650 | Optim: 19650 | Time: 1.90s | LR: 0.000333 | SkipOpt: 0 loss: 0.229712 | l2_loss: 0.019575 | psnr: 17.083061 | perceptual_loss: 0.420275 | norm_perceptual_loss: 21.470373 [09:14:26.540537] [Epoch 0] | Iter: 19700 | Optim: 19700 | Time: 1.90s | LR: 0.000333 | SkipOpt: 0 loss: 0.221340 | l2_loss: 0.015795 | psnr: 18.014830 | perceptual_loss: 0.411090 | norm_perceptual_loss: 26.026737 [09:15:23.707327] [Epoch 0] | Iter: 19750 | Optim: 19750 | Time: 0.75s | LR: 0.000332 | SkipOpt: 0 loss: 0.168020 | l2_loss: 0.008642 | psnr: 20.633795 | perceptual_loss: 0.318755 | norm_perceptual_loss: 36.883846 [09:16:24.031982] [Epoch 0] | Iter: 19800 | Optim: 19800 | Time: 1.91s | LR: 0.000332 | SkipOpt: 0 loss: 0.236225 | l2_loss: 0.015955 | psnr: 17.970900 | perceptual_loss: 0.440539 | norm_perceptual_loss: 27.610506 [09:17:23.979001] [Epoch 0] | Iter: 19850 | Optim: 19850 | Time: 1.08s | LR: 0.000331 | SkipOpt: 0 loss: 0.219872 | l2_loss: 0.012843 | psnr: 18.913256 | perceptual_loss: 0.414057 | norm_perceptual_loss: 32.239285 [09:18:22.111499] [Epoch 0] | Iter: 19900 | Optim: 19900 | Time: 0.75s | LR: 0.000331 | SkipOpt: 0 loss: 0.191910 | l2_loss: 0.007595 | psnr: 21.194981 | perceptual_loss: 0.368632 | norm_perceptual_loss: 48.538994 [09:19:23.102595] [Epoch 0] | Iter: 19950 | Optim: 19950 | Time: 1.07s | LR: 0.000331 | SkipOpt: 0 loss: 0.229568 | l2_loss: 0.014741 | psnr: 18.314844 | perceptual_loss: 0.429654 | norm_perceptual_loss: 29.147654 [09:20:22.486134] [Epoch 0] | Iter: 20000 | Optim: 20000 | Time: 1.07s | LR: 0.000330 | SkipOpt: 0 loss: 0.204643 | l2_loss: 0.012868 | psnr: 18.904837 | perceptual_loss: 0.383549 | norm_perceptual_loss: 29.806034 [09:20:24.201655] Found 140 sequences [09:20:49.260516] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [09:20:50.115243] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [09:21:41.428220] Gathering scores from all processes... [09:21:41.429142] Padding local tensor of length 140 to max count: 140 [09:21:41.429718] Padding local tensor of length 140 to max count: 140 [09:21:41.430197] Padding local tensor of length 140 to max count: 140 [09:21:41.430466] Gathering scene names from all processes... [09:21:41.430584] Padding local tensor of length 140 on rank 0 to max count: 140 [09:21:41.431623] Evaluation metrics: [09:21:41.431798] psnr: 16.4638 [09:21:41.431878] ssim: 0.4485 [09:21:41.431964] lpips: 0.5485 [09:21:41.435501] Found 140 sequences [09:22:05.857718] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [09:22:06.707284] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [09:22:59.307659] Gathering scores from all processes... [09:22:59.308820] Padding local tensor of length 140 to max count: 140 [09:22:59.309466] Padding local tensor of length 140 to max count: 140 [09:22:59.309962] Padding local tensor of length 140 to max count: 140 [09:22:59.310245] Gathering scene names from all processes... [09:22:59.310352] Padding local tensor of length 140 on rank 0 to max count: 140 [09:22:59.311340] Evaluation metrics: [09:22:59.311508] psnr: 18.7191 [09:22:59.311592] ssim: 0.5100 [09:22:59.311669] lpips: 0.4704 [09:22:59.315140] Found 140 sequences [09:23:25.477111] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [09:23:26.322741] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [09:24:24.148484] Gathering scores from all processes... [09:24:24.149435] Padding local tensor of length 140 to max count: 140 [09:24:24.150084] Padding local tensor of length 140 to max count: 140 [09:24:24.150616] Padding local tensor of length 140 to max count: 140 [09:24:24.150928] Gathering scene names from all processes... [09:24:24.151070] Padding local tensor of length 140 on rank 0 to max count: 140 [09:24:24.152239] Evaluation metrics: [09:24:24.152429] psnr: 19.6281 [09:24:24.152521] ssim: 0.5369 [09:24:24.152605] lpips: 0.4422 [09:24:24.154098] [Eval][iter 20000] dl3dv_2view | psnr: 16.463772 | ssim: 0.448511 | lpips: 0.548510 || dl3dv_4view | psnr: 18.719131 | ssim: 0.510030 | lpips: 0.470424 || dl3dv_6view | psnr: 19.628149 | ssim: 0.536896 | lpips: 0.442171 [09:24:24.156907] [Eval][iter 20000] dl3dv_2view | psnr: 16.463772 | ssim: 0.448511 | lpips: 0.548510 || dl3dv_4view | psnr: 18.719131 | ssim: 0.510030 | lpips: 0.470424 || dl3dv_6view | psnr: 19.628149 | ssim: 0.536896 | lpips: 0.442171 [09:24:28.745068] Saved checkpoint at iteration 20000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [09:24:28.745440] Saved latest checkpoint at iteration 20000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [09:25:33.145881] [Epoch 0] | Iter: 20050 | Optim: 20050 | Time: 1.91s | LR: 0.000330 | SkipOpt: 0 loss: 0.211336 | l2_loss: 0.014823 | psnr: 18.290628 | perceptual_loss: 0.393026 | norm_perceptual_loss: 26.514557 [09:26:32.952883] [Epoch 0] | Iter: 20100 | Optim: 20100 | Time: 1.91s | LR: 0.000330 | SkipOpt: 0 loss: 0.243872 | l2_loss: 0.018531 | psnr: 17.321012 | perceptual_loss: 0.450681 | norm_perceptual_loss: 24.320398 [09:27:31.614363] [Epoch 0] | Iter: 20150 | Optim: 20150 | Time: 1.07s | LR: 0.000329 | SkipOpt: 0 loss: 0.226673 | l2_loss: 0.012584 | psnr: 19.001678 | perceptual_loss: 0.428177 | norm_perceptual_loss: 34.024433 [09:28:35.886540] [Epoch 0] | Iter: 20200 | Optim: 20200 | Time: 0.75s | LR: 0.000329 | SkipOpt: 0 loss: 0.195719 | l2_loss: 0.010162 | psnr: 19.930346 | perceptual_loss: 0.371114 | norm_perceptual_loss: 36.520954 [09:29:36.656582] [Epoch 0] | Iter: 20250 | Optim: 20250 | Time: 0.85s | LR: 0.000328 | SkipOpt: 0 loss: 0.264337 | l2_loss: 0.021618 | psnr: 16.651751 | perceptual_loss: 0.485437 | norm_perceptual_loss: 22.454739 [09:30:36.527609] [Epoch 0] | Iter: 20300 | Optim: 20300 | Time: 1.27s | LR: 0.000328 | SkipOpt: 0 loss: 0.228387 | l2_loss: 0.015506 | psnr: 18.095078 | perceptual_loss: 0.425762 | norm_perceptual_loss: 27.458363 [09:31:33.395501] [Epoch 0] | Iter: 20350 | Optim: 20350 | Time: 0.75s | LR: 0.000328 | SkipOpt: 0 loss: 0.212561 | l2_loss: 0.012408 | psnr: 19.063103 | perceptual_loss: 0.400308 | norm_perceptual_loss: 32.262943 [09:32:35.169524] [Epoch 0] | Iter: 20400 | Optim: 20400 | Time: 1.08s | LR: 0.000327 | SkipOpt: 0 loss: 0.214446 | l2_loss: 0.016308 | psnr: 17.875935 | perceptual_loss: 0.396276 | norm_perceptual_loss: 24.299189 [09:33:40.477422] [Epoch 0] | Iter: 20450 | Optim: 20450 | Time: 1.91s | LR: 0.000327 | SkipOpt: 0 loss: 0.223390 | l2_loss: 0.017977 | psnr: 17.452881 | perceptual_loss: 0.410827 | norm_perceptual_loss: 22.853182 [09:34:42.281961] [Epoch 0] | Iter: 20500 | Optim: 20500 | Time: 1.91s | LR: 0.000326 | SkipOpt: 0 loss: 0.249214 | l2_loss: 0.018489 | psnr: 17.330902 | perceptual_loss: 0.461450 | norm_perceptual_loss: 24.958271 [09:35:40.612847] [Epoch 0] | Iter: 20550 | Optim: 20550 | Time: 0.75s | LR: 0.000326 | SkipOpt: 0 loss: 0.194797 | l2_loss: 0.012307 | psnr: 19.098356 | perceptual_loss: 0.364978 | norm_perceptual_loss: 29.655327 [09:36:45.203501] [Epoch 0] | Iter: 20600 | Optim: 20600 | Time: 1.90s | LR: 0.000326 | SkipOpt: 0 loss: 0.222824 | l2_loss: 0.017445 | psnr: 17.583210 | perceptual_loss: 0.410757 | norm_perceptual_loss: 23.545387 [09:37:48.998321] [Epoch 0] | Iter: 20650 | Optim: 20650 | Time: 0.75s | LR: 0.000325 | SkipOpt: 0 loss: 0.192826 | l2_loss: 0.009168 | psnr: 20.377079 | perceptual_loss: 0.367315 | norm_perceptual_loss: 40.063267 [09:38:52.532736] [Epoch 0] | Iter: 20700 | Optim: 20700 | Time: 0.85s | LR: 0.000325 | SkipOpt: 0 loss: 0.174655 | l2_loss: 0.008497 | psnr: 20.707588 | perceptual_loss: 0.332318 | norm_perceptual_loss: 39.112179 [09:39:54.761636] [Epoch 0] | Iter: 20750 | Optim: 20750 | Time: 0.85s | LR: 0.000324 | SkipOpt: 0 loss: 0.234756 | l2_loss: 0.019607 | psnr: 17.075979 | perceptual_loss: 0.430298 | norm_perceptual_loss: 21.946606 [09:40:29.525494] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [09:40:55.994667] [Epoch 0] | Iter: 20800 | Optim: 20800 | Time: 1.26s | LR: 0.000324 | SkipOpt: 0 loss: 0.192771 | l2_loss: 0.012949 | psnr: 18.877762 | perceptual_loss: 0.359644 | norm_perceptual_loss: 27.774656 [09:41:58.484175] [Epoch 0] | Iter: 20850 | Optim: 20850 | Time: 1.90s | LR: 0.000324 | SkipOpt: 0 loss: 0.221452 | l2_loss: 0.016456 | psnr: 17.836826 | perceptual_loss: 0.409992 | norm_perceptual_loss: 24.914835 [09:43:00.630573] [Epoch 0] | Iter: 20900 | Optim: 20900 | Time: 0.75s | LR: 0.000323 | SkipOpt: 0 loss: 0.194971 | l2_loss: 0.014375 | psnr: 18.423986 | perceptual_loss: 0.361193 | norm_perceptual_loss: 25.126837 [09:43:59.685114] [Epoch 0] | Iter: 20950 | Optim: 20950 | Time: 0.75s | LR: 0.000323 | SkipOpt: 0 loss: 0.228042 | l2_loss: 0.014497 | psnr: 18.387173 | perceptual_loss: 0.427089 | norm_perceptual_loss: 29.460175 [09:44:53.044629] [Epoch 0] | Iter: 21000 | Optim: 21000 | Time: 0.75s | LR: 0.000322 | SkipOpt: 0 loss: 0.212354 | l2_loss: 0.012781 | psnr: 18.934519 | perceptual_loss: 0.399148 | norm_perceptual_loss: 31.230972 [09:44:58.095325] Saved checkpoint at iteration 21000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [09:44:58.096146] Saved latest checkpoint at iteration 21000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [09:45:58.796668] [Epoch 0] | Iter: 21050 | Optim: 21050 | Time: 1.06s | LR: 0.000322 | SkipOpt: 0 loss: 0.207091 | l2_loss: 0.015150 | psnr: 18.195984 | perceptual_loss: 0.383882 | norm_perceptual_loss: 25.339382 [09:46:48.110177] New dataloader... [09:46:48.111314] Building dynamic dataloader with seed: 0 [09:46:48.185058] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.185219] Worker seed: 12346 [09:46:48.238682] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.238833] Worker seed: 12347 [09:46:48.275656] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.275809] Worker seed: 12348 [09:46:48.302375] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.302482] Worker seed: 12349 [09:46:48.348183] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.348382] Worker seed: 12350 [09:46:48.369006] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.369165] Worker seed: 12351 [09:46:48.399674] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.399824] Worker seed: 12352 [09:46:48.430343] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.430494] Worker seed: 12353 [09:46:48.461016] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.461169] Worker seed: 12354 [09:46:48.490582] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.490726] Worker seed: 12355 [09:46:48.522484] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.522623] Worker seed: 12356 [09:46:48.553178] Rank: 0, World size: 1, Distributed rank: 0 [09:46:48.553329] Worker seed: 12357 [09:46:59.094032] [Epoch 1] | Iter: 21100 | Optim: 21100 | Time: 0.85s | LR: 0.000322 | SkipOpt: 0 loss: 0.197412 | l2_loss: 0.013052 | psnr: 18.843216 | perceptual_loss: 0.368720 | norm_perceptual_loss: 28.249985 [09:47:53.208967] [Epoch 1] | Iter: 21150 | Optim: 21150 | Time: 0.85s | LR: 0.000321 | SkipOpt: 0 loss: 0.219636 | l2_loss: 0.012908 | psnr: 18.891518 | perceptual_loss: 0.413457 | norm_perceptual_loss: 32.031864 [09:48:53.784077] [Epoch 1] | Iter: 21200 | Optim: 21200 | Time: 1.91s | LR: 0.000321 | SkipOpt: 0 loss: 0.237503 | l2_loss: 0.018124 | psnr: 17.417534 | perceptual_loss: 0.438758 | norm_perceptual_loss: 24.209074 [09:49:57.706505] [Epoch 1] | Iter: 21250 | Optim: 21250 | Time: 1.91s | LR: 0.000320 | SkipOpt: 0 loss: 0.225245 | l2_loss: 0.015698 | psnr: 18.041527 | perceptual_loss: 0.419095 | norm_perceptual_loss: 26.697134 [09:51:00.639018] [Epoch 1] | Iter: 21300 | Optim: 21300 | Time: 1.91s | LR: 0.000320 | SkipOpt: 0 loss: 0.241938 | l2_loss: 0.018439 | psnr: 17.342709 | perceptual_loss: 0.446999 | norm_perceptual_loss: 24.242487 [09:51:53.613780] [Epoch 1] | Iter: 21350 | Optim: 21350 | Time: 0.85s | LR: 0.000320 | SkipOpt: 0 loss: 0.199248 | l2_loss: 0.011810 | psnr: 19.277344 | perceptual_loss: 0.374875 | norm_perceptual_loss: 31.741030 [09:52:52.294142] [Epoch 1] | Iter: 21400 | Optim: 21400 | Time: 0.75s | LR: 0.000319 | SkipOpt: 0 loss: 0.205446 | l2_loss: 0.010303 | psnr: 19.870235 | perceptual_loss: 0.390286 | norm_perceptual_loss: 37.879650 [09:53:56.782910] [Epoch 1] | Iter: 21450 | Optim: 21450 | Time: 1.07s | LR: 0.000319 | SkipOpt: 0 loss: 0.161453 | l2_loss: 0.009235 | psnr: 20.345772 | perceptual_loss: 0.304436 | norm_perceptual_loss: 32.966476 [09:55:03.008705] [Epoch 1] | Iter: 21500 | Optim: 21500 | Time: 1.08s | LR: 0.000318 | SkipOpt: 0 loss: 0.195109 | l2_loss: 0.012839 | psnr: 18.914600 | perceptual_loss: 0.364540 | norm_perceptual_loss: 28.392578 [09:56:03.707878] [Epoch 1] | Iter: 21550 | Optim: 21550 | Time: 0.75s | LR: 0.000318 | SkipOpt: 0 loss: 0.198331 | l2_loss: 0.014341 | psnr: 18.434122 | perceptual_loss: 0.367978 | norm_perceptual_loss: 25.658689 [09:57:04.379016] [Epoch 1] | Iter: 21600 | Optim: 21600 | Time: 0.75s | LR: 0.000318 | SkipOpt: 0 loss: 0.186675 | l2_loss: 0.009700 | psnr: 20.132105 | perceptual_loss: 0.353949 | norm_perceptual_loss: 36.488064 [09:58:03.689996] [Epoch 1] | Iter: 21650 | Optim: 21650 | Time: 1.07s | LR: 0.000317 | SkipOpt: 0 loss: 0.246287 | l2_loss: 0.018999 | psnr: 17.212618 | perceptual_loss: 0.454575 | norm_perceptual_loss: 23.925852 [09:59:03.009605] [Epoch 1] | Iter: 21700 | Optim: 21700 | Time: 1.90s | LR: 0.000317 | SkipOpt: 0 loss: 0.245131 | l2_loss: 0.020754 | psnr: 16.829031 | perceptual_loss: 0.448754 | norm_perceptual_loss: 21.622789 [10:00:08.415955] [Epoch 1] | Iter: 21750 | Optim: 21750 | Time: 1.91s | LR: 0.000316 | SkipOpt: 0 loss: 0.267540 | l2_loss: 0.022772 | psnr: 16.425938 | perceptual_loss: 0.489536 | norm_perceptual_loss: 21.497011 [10:01:09.871825] [Epoch 1] | Iter: 21800 | Optim: 21800 | Time: 1.90s | LR: 0.000316 | SkipOpt: 0 loss: 0.215161 | l2_loss: 0.015172 | psnr: 18.189657 | perceptual_loss: 0.399978 | norm_perceptual_loss: 26.363440 [10:01:38.757528] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [10:02:11.358361] [Epoch 1] | Iter: 21850 | Optim: 21850 | Time: 1.07s | LR: 0.000316 | SkipOpt: 0 loss: 0.174164 | l2_loss: 0.008932 | psnr: 20.490549 | perceptual_loss: 0.330463 | norm_perceptual_loss: 36.997997 [10:03:17.490933] [Epoch 1] | Iter: 21900 | Optim: 21900 | Time: 0.75s | LR: 0.000315 | SkipOpt: 0 loss: 0.196436 | l2_loss: 0.010288 | psnr: 19.876554 | perceptual_loss: 0.372295 | norm_perceptual_loss: 36.186226 [10:04:19.631301] [Epoch 1] | Iter: 21950 | Optim: 21950 | Time: 1.91s | LR: 0.000315 | SkipOpt: 0 loss: 0.253125 | l2_loss: 0.018661 | psnr: 17.290634 | perceptual_loss: 0.468927 | norm_perceptual_loss: 25.128624 [10:05:21.065941] [Epoch 1] | Iter: 22000 | Optim: 22000 | Time: 0.75s | LR: 0.000314 | SkipOpt: 0 loss: 0.230793 | l2_loss: 0.013377 | psnr: 18.736252 | perceptual_loss: 0.434832 | norm_perceptual_loss: 32.504704 [10:05:27.449167] Saved checkpoint at iteration 22000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [10:05:27.449635] Saved latest checkpoint at iteration 22000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [10:06:30.929909] [Epoch 1] | Iter: 22050 | Optim: 22050 | Time: 1.91s | LR: 0.000314 | SkipOpt: 0 loss: 0.208627 | l2_loss: 0.015740 | psnr: 18.030092 | perceptual_loss: 0.385775 | norm_perceptual_loss: 24.509998 [10:07:35.458230] [Epoch 1] | Iter: 22100 | Optim: 22100 | Time: 1.07s | LR: 0.000313 | SkipOpt: 0 loss: 0.190861 | l2_loss: 0.010239 | psnr: 19.897577 | perceptual_loss: 0.361245 | norm_perceptual_loss: 35.282505 [10:08:39.717651] [Epoch 1] | Iter: 22150 | Optim: 22150 | Time: 1.07s | LR: 0.000313 | SkipOpt: 0 loss: 0.192528 | l2_loss: 0.010725 | psnr: 19.696207 | perceptual_loss: 0.363606 | norm_perceptual_loss: 33.904095 [10:09:40.814448] [Epoch 1] | Iter: 22200 | Optim: 22200 | Time: 1.07s | LR: 0.000313 | SkipOpt: 0 loss: 0.208530 | l2_loss: 0.014342 | psnr: 18.434046 | perceptual_loss: 0.388376 | norm_perceptual_loss: 27.080538 [10:10:42.517388] [Epoch 1] | Iter: 22250 | Optim: 22250 | Time: 1.07s | LR: 0.000312 | SkipOpt: 0 loss: 0.247220 | l2_loss: 0.021558 | psnr: 16.663979 | perceptual_loss: 0.451324 | norm_perceptual_loss: 20.935629 [10:11:45.682003] [Epoch 1] | Iter: 22300 | Optim: 22300 | Time: 1.91s | LR: 0.000312 | SkipOpt: 0 loss: 0.207808 | l2_loss: 0.014402 | psnr: 18.415758 | perceptual_loss: 0.386811 | norm_perceptual_loss: 26.858051 [10:12:51.116746] [Epoch 1] | Iter: 22350 | Optim: 22350 | Time: 1.07s | LR: 0.000311 | SkipOpt: 0 loss: 0.194230 | l2_loss: 0.012803 | psnr: 18.926781 | perceptual_loss: 0.362853 | norm_perceptual_loss: 28.340614 [10:13:54.521614] [Epoch 1] | Iter: 22400 | Optim: 22400 | Time: 1.07s | LR: 0.000311 | SkipOpt: 0 loss: 0.209415 | l2_loss: 0.012402 | psnr: 19.065161 | perceptual_loss: 0.394026 | norm_perceptual_loss: 31.771742 [10:14:52.925784] [Epoch 1] | Iter: 22450 | Optim: 22450 | Time: 0.75s | LR: 0.000310 | SkipOpt: 0 loss: 0.184063 | l2_loss: 0.010635 | psnr: 19.732746 | perceptual_loss: 0.346856 | norm_perceptual_loss: 32.615513 [10:15:54.138058] [Epoch 1] | Iter: 22500 | Optim: 22500 | Time: 0.75s | LR: 0.000310 | SkipOpt: 0 loss: 0.208336 | l2_loss: 0.011872 | psnr: 19.254816 | perceptual_loss: 0.392929 | norm_perceptual_loss: 33.097549 [10:16:55.089343] [Epoch 1] | Iter: 22550 | Optim: 22550 | Time: 0.75s | LR: 0.000310 | SkipOpt: 0 loss: 0.206843 | l2_loss: 0.013987 | psnr: 18.542858 | perceptual_loss: 0.385713 | norm_perceptual_loss: 27.577150 [10:18:05.007631] [Epoch 1] | Iter: 22600 | Optim: 22600 | Time: 1.91s | LR: 0.000309 | SkipOpt: 0 loss: 0.243374 | l2_loss: 0.019805 | psnr: 17.032297 | perceptual_loss: 0.447139 | norm_perceptual_loss: 22.577326 [10:19:12.472910] [Epoch 1] | Iter: 22650 | Optim: 22650 | Time: 1.91s | LR: 0.000309 | SkipOpt: 0 loss: 0.233365 | l2_loss: 0.017544 | psnr: 17.558830 | perceptual_loss: 0.431644 | norm_perceptual_loss: 24.604136 [10:20:12.001562] [Epoch 1] | Iter: 22700 | Optim: 22700 | Time: 1.07s | LR: 0.000308 | SkipOpt: 0 loss: 0.171949 | l2_loss: 0.010233 | psnr: 19.900082 | perceptual_loss: 0.323433 | norm_perceptual_loss: 31.607702 [10:21:14.172063] [Epoch 1] | Iter: 22750 | Optim: 22750 | Time: 0.75s | LR: 0.000308 | SkipOpt: 0 loss: 0.194499 | l2_loss: 0.010371 | psnr: 19.841805 | perceptual_loss: 0.368255 | norm_perceptual_loss: 35.508255 [10:22:17.879986] [Epoch 1] | Iter: 22800 | Optim: 22800 | Time: 1.91s | LR: 0.000308 | SkipOpt: 0 loss: 0.236668 | l2_loss: 0.019877 | psnr: 17.016388 | perceptual_loss: 0.433580 | norm_perceptual_loss: 21.812643 [10:23:22.021306] [Epoch 1] | Iter: 22850 | Optim: 22850 | Time: 1.90s | LR: 0.000307 | SkipOpt: 0 loss: 0.211666 | l2_loss: 0.014357 | psnr: 18.429436 | perceptual_loss: 0.394619 | norm_perceptual_loss: 27.486609 [10:24:26.357130] [Epoch 1] | Iter: 22900 | Optim: 22900 | Time: 0.75s | LR: 0.000307 | SkipOpt: 0 loss: 0.188589 | l2_loss: 0.010681 | psnr: 19.713816 | perceptual_loss: 0.355815 | norm_perceptual_loss: 33.312401 [10:25:24.320164] [Epoch 1] | Iter: 22950 | Optim: 22950 | Time: 0.75s | LR: 0.000306 | SkipOpt: 0 loss: 0.144783 | l2_loss: 0.005906 | psnr: 22.287296 | perceptual_loss: 0.277755 | norm_perceptual_loss: 47.031719 [10:26:25.918235] [Epoch 1] | Iter: 23000 | Optim: 23000 | Time: 1.26s | LR: 0.000306 | SkipOpt: 0 loss: 0.223310 | l2_loss: 0.015445 | psnr: 18.112059 | perceptual_loss: 0.415729 | norm_perceptual_loss: 26.916372 [10:26:31.299546] Saved checkpoint at iteration 23000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [10:26:31.299797] Saved latest checkpoint at iteration 23000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [10:27:31.979876] [Epoch 1] | Iter: 23050 | Optim: 23050 | Time: 1.91s | LR: 0.000305 | SkipOpt: 0 loss: 0.221848 | l2_loss: 0.016161 | psnr: 17.915218 | perceptual_loss: 0.411372 | norm_perceptual_loss: 25.454048 [10:28:34.482029] [Epoch 1] | Iter: 23100 | Optim: 23100 | Time: 1.90s | LR: 0.000305 | SkipOpt: 0 loss: 0.219047 | l2_loss: 0.015300 | psnr: 18.153187 | perceptual_loss: 0.407494 | norm_perceptual_loss: 26.634197 [10:29:37.204611] [Epoch 1] | Iter: 23150 | Optim: 23150 | Time: 1.07s | LR: 0.000305 | SkipOpt: 0 loss: 0.201850 | l2_loss: 0.013860 | psnr: 18.582291 | perceptual_loss: 0.375979 | norm_perceptual_loss: 27.126421 [10:30:37.314970] [Epoch 1] | Iter: 23200 | Optim: 23200 | Time: 1.27s | LR: 0.000304 | SkipOpt: 0 loss: 0.230323 | l2_loss: 0.017595 | psnr: 17.546177 | perceptual_loss: 0.425456 | norm_perceptual_loss: 24.180870 [10:31:48.933301] [Epoch 1] | Iter: 23250 | Optim: 23250 | Time: 0.85s | LR: 0.000304 | SkipOpt: 0 loss: 0.179928 | l2_loss: 0.007231 | psnr: 21.407825 | perceptual_loss: 0.345394 | norm_perceptual_loss: 47.763580 [10:32:51.254809] [Epoch 1] | Iter: 23300 | Optim: 23300 | Time: 1.91s | LR: 0.000303 | SkipOpt: 0 loss: 0.222309 | l2_loss: 0.017261 | psnr: 17.629387 | perceptual_loss: 0.410097 | norm_perceptual_loss: 23.758841 [10:33:55.412330] [Epoch 1] | Iter: 23350 | Optim: 23350 | Time: 0.85s | LR: 0.000303 | SkipOpt: 0 loss: 0.213609 | l2_loss: 0.014510 | psnr: 18.383251 | perceptual_loss: 0.398198 | norm_perceptual_loss: 27.442547 [10:34:54.811420] [Epoch 1] | Iter: 23400 | Optim: 23400 | Time: 1.07s | LR: 0.000302 | SkipOpt: 0 loss: 0.200071 | l2_loss: 0.013053 | psnr: 18.842842 | perceptual_loss: 0.374036 | norm_perceptual_loss: 28.654774 [10:35:55.775484] [Epoch 1] | Iter: 23450 | Optim: 23450 | Time: 1.07s | LR: 0.000302 | SkipOpt: 0 loss: 0.236073 | l2_loss: 0.014160 | psnr: 18.489374 | perceptual_loss: 0.443827 | norm_perceptual_loss: 31.343744 [10:37:00.614831] [Epoch 1] | Iter: 23500 | Optim: 23500 | Time: 0.75s | LR: 0.000301 | SkipOpt: 0 loss: 0.176156 | l2_loss: 0.008522 | psnr: 20.694645 | perceptual_loss: 0.335268 | norm_perceptual_loss: 39.342060 [10:38:01.328685] [Epoch 1] | Iter: 23550 | Optim: 23550 | Time: 1.91s | LR: 0.000301 | SkipOpt: 0 loss: 0.222315 | l2_loss: 0.015648 | psnr: 18.055279 | perceptual_loss: 0.413333 | norm_perceptual_loss: 26.413641 [10:39:05.805323] [Epoch 1] | Iter: 23600 | Optim: 23600 | Time: 1.27s | LR: 0.000301 | SkipOpt: 0 loss: 0.239713 | l2_loss: 0.017920 | psnr: 17.466516 | perceptual_loss: 0.443584 | norm_perceptual_loss: 24.752996 [10:40:06.095703] [Epoch 1] | Iter: 23650 | Optim: 23650 | Time: 1.27s | LR: 0.000300 | SkipOpt: 0 loss: 0.219501 | l2_loss: 0.013926 | psnr: 18.561871 | perceptual_loss: 0.411151 | norm_perceptual_loss: 29.524876 [10:41:15.367198] [Epoch 1] | Iter: 23700 | Optim: 23700 | Time: 1.26s | LR: 0.000300 | SkipOpt: 0 loss: 0.203195 | l2_loss: 0.017303 | psnr: 17.618683 | perceptual_loss: 0.371783 | norm_perceptual_loss: 21.486094 [10:42:26.629621] [Epoch 1] | Iter: 23750 | Optim: 23750 | Time: 1.26s | LR: 0.000299 | SkipOpt: 0 loss: 0.248984 | l2_loss: 0.016870 | psnr: 17.728912 | perceptual_loss: 0.464228 | norm_perceptual_loss: 27.518368 [10:43:30.204874] [Epoch 1] | Iter: 23800 | Optim: 23800 | Time: 0.75s | LR: 0.000299 | SkipOpt: 0 loss: 0.194026 | l2_loss: 0.008959 | psnr: 20.477428 | perceptual_loss: 0.370133 | norm_perceptual_loss: 41.314377 [10:44:31.550479] [Epoch 1] | Iter: 23850 | Optim: 23850 | Time: 0.75s | LR: 0.000298 | SkipOpt: 0 loss: 0.220463 | l2_loss: 0.013707 | psnr: 18.630659 | perceptual_loss: 0.413512 | norm_perceptual_loss: 30.168499 [10:45:33.459133] [Epoch 1] | Iter: 23900 | Optim: 23900 | Time: 1.07s | LR: 0.000298 | SkipOpt: 0 loss: 0.217881 | l2_loss: 0.015565 | psnr: 18.078489 | perceptual_loss: 0.404632 | norm_perceptual_loss: 25.996136 [10:46:33.381328] [Epoch 1] | Iter: 23950 | Optim: 23950 | Time: 1.07s | LR: 0.000298 | SkipOpt: 0 loss: 0.212513 | l2_loss: 0.016657 | psnr: 17.784050 | perceptual_loss: 0.391712 | norm_perceptual_loss: 23.516449 [10:47:40.956317] [Epoch 1] | Iter: 24000 | Optim: 24000 | Time: 1.91s | LR: 0.000297 | SkipOpt: 0 loss: 0.243674 | l2_loss: 0.022286 | psnr: 16.519600 | perceptual_loss: 0.442774 | norm_perceptual_loss: 19.867470 [10:47:47.324326] Saved checkpoint at iteration 24000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [10:47:47.324743] Saved latest checkpoint at iteration 24000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [10:48:45.114623] [Epoch 1] | Iter: 24050 | Optim: 24050 | Time: 0.86s | LR: 0.000297 | SkipOpt: 0 loss: 0.196918 | l2_loss: 0.012092 | psnr: 19.174944 | perceptual_loss: 0.369651 | norm_perceptual_loss: 30.569368 [10:49:46.601943] [Epoch 1] | Iter: 24100 | Optim: 24100 | Time: 0.75s | LR: 0.000296 | SkipOpt: 0 loss: 0.186796 | l2_loss: 0.009227 | psnr: 20.349552 | perceptual_loss: 0.355139 | norm_perceptual_loss: 38.490475 [10:50:46.437233] [Epoch 1] | Iter: 24150 | Optim: 24150 | Time: 0.85s | LR: 0.000296 | SkipOpt: 0 loss: 0.185774 | l2_loss: 0.010785 | psnr: 19.671707 | perceptual_loss: 0.349977 | norm_perceptual_loss: 32.449627 [10:51:49.279566] [Epoch 1] | Iter: 24200 | Optim: 24200 | Time: 1.07s | LR: 0.000295 | SkipOpt: 0 loss: 0.232464 | l2_loss: 0.017257 | psnr: 17.630299 | perceptual_loss: 0.430413 | norm_perceptual_loss: 24.941057 [10:52:47.141072] [Epoch 1] | Iter: 24250 | Optim: 24250 | Time: 1.91s | LR: 0.000295 | SkipOpt: 0 loss: 0.219909 | l2_loss: 0.018033 | psnr: 17.439322 | perceptual_loss: 0.403752 | norm_perceptual_loss: 22.389639 [10:53:53.835093] [Epoch 1] | Iter: 24300 | Optim: 24300 | Time: 1.91s | LR: 0.000294 | SkipOpt: 0 loss: 0.219225 | l2_loss: 0.015385 | psnr: 18.129019 | perceptual_loss: 0.407679 | norm_perceptual_loss: 26.498436 [10:54:53.228441] [Epoch 1] | Iter: 24350 | Optim: 24350 | Time: 1.07s | LR: 0.000294 | SkipOpt: 0 loss: 0.243772 | l2_loss: 0.021035 | psnr: 16.770523 | perceptual_loss: 0.445474 | norm_perceptual_loss: 21.177509 [10:55:52.683774] [Epoch 1] | Iter: 24400 | Optim: 24400 | Time: 1.91s | LR: 0.000293 | SkipOpt: 0 loss: 0.214943 | l2_loss: 0.018275 | psnr: 17.381435 | perceptual_loss: 0.393337 | norm_perceptual_loss: 21.523266 [10:56:52.746590] [Epoch 1] | Iter: 24450 | Optim: 24450 | Time: 1.91s | LR: 0.000293 | SkipOpt: 0 loss: 0.250401 | l2_loss: 0.018711 | psnr: 17.279003 | perceptual_loss: 0.463380 | norm_perceptual_loss: 24.764938 [10:57:55.558729] [Epoch 1] | Iter: 24500 | Optim: 24500 | Time: 0.87s | LR: 0.000293 | SkipOpt: 0 loss: 0.186382 | l2_loss: 0.013054 | psnr: 18.842714 | perceptual_loss: 0.346656 | norm_perceptual_loss: 26.556471 [10:59:00.619741] [Epoch 1] | Iter: 24550 | Optim: 24550 | Time: 0.75s | LR: 0.000292 | SkipOpt: 0 loss: 0.154563 | l2_loss: 0.006895 | psnr: 21.614605 | perceptual_loss: 0.295336 | norm_perceptual_loss: 42.832829 [10:59:59.973725] [Epoch 1] | Iter: 24600 | Optim: 24600 | Time: 0.75s | LR: 0.000292 | SkipOpt: 0 loss: 0.154377 | l2_loss: 0.006952 | psnr: 21.578959 | perceptual_loss: 0.294850 | norm_perceptual_loss: 42.412731 [11:00:55.051553] [Epoch 1] | Iter: 24650 | Optim: 24650 | Time: 1.90s | LR: 0.000291 | SkipOpt: 0 loss: 0.240079 | l2_loss: 0.017159 | psnr: 17.654972 | perceptual_loss: 0.445839 | norm_perceptual_loss: 25.982143 [11:01:57.452761] [Epoch 1] | Iter: 24700 | Optim: 24700 | Time: 1.09s | LR: 0.000291 | SkipOpt: 0 loss: 0.175273 | l2_loss: 0.010718 | psnr: 19.698956 | perceptual_loss: 0.329111 | norm_perceptual_loss: 30.707056 [11:02:58.325153] [Epoch 1] | Iter: 24750 | Optim: 24750 | Time: 1.08s | LR: 0.000290 | SkipOpt: 0 loss: 0.241877 | l2_loss: 0.019246 | psnr: 17.156578 | perceptual_loss: 0.445263 | norm_perceptual_loss: 23.135239 [11:04:01.589665] [Epoch 1] | Iter: 24800 | Optim: 24800 | Time: 1.07s | LR: 0.000290 | SkipOpt: 0 loss: 0.253569 | l2_loss: 0.022080 | psnr: 16.559999 | perceptual_loss: 0.462978 | norm_perceptual_loss: 20.968134 [11:05:03.973850] [Epoch 1] | Iter: 24850 | Optim: 24850 | Time: 1.07s | LR: 0.000289 | SkipOpt: 0 loss: 0.274556 | l2_loss: 0.021488 | psnr: 16.678064 | perceptual_loss: 0.506137 | norm_perceptual_loss: 23.554516 [11:06:00.395043] [Epoch 1] | Iter: 24900 | Optim: 24900 | Time: 1.07s | LR: 0.000289 | SkipOpt: 0 loss: 0.199639 | l2_loss: 0.010155 | psnr: 19.933292 | perceptual_loss: 0.378969 | norm_perceptual_loss: 37.319267 [11:07:07.355778] [Epoch 1] | Iter: 24950 | Optim: 24950 | Time: 0.76s | LR: 0.000289 | SkipOpt: 0 loss: 0.171592 | l2_loss: 0.012781 | psnr: 18.934189 | perceptual_loss: 0.317620 | norm_perceptual_loss: 24.850029 [11:08:02.799809] [Epoch 1] | Iter: 25000 | Optim: 25000 | Time: 0.75s | LR: 0.000288 | SkipOpt: 0 loss: 0.196866 | l2_loss: 0.011881 | psnr: 19.251532 | perceptual_loss: 0.369971 | norm_perceptual_loss: 31.140152 [11:08:04.445293] Found 140 sequences [11:08:28.098701] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [11:08:28.966595] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [11:09:20.257144] Gathering scores from all processes... [11:09:20.258461] Padding local tensor of length 140 to max count: 140 [11:09:20.259322] Padding local tensor of length 140 to max count: 140 [11:09:20.259884] Padding local tensor of length 140 to max count: 140 [11:09:20.260239] Gathering scene names from all processes... [11:09:20.260369] Padding local tensor of length 140 on rank 0 to max count: 140 [11:09:20.261503] Evaluation metrics: [11:09:20.261712] psnr: 16.8583 [11:09:20.261813] ssim: 0.4643 [11:09:20.261912] lpips: 0.5332 [11:09:20.265554] Found 140 sequences [11:09:44.521609] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [11:09:45.523349] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [11:10:38.143770] Gathering scores from all processes... [11:10:38.144609] Padding local tensor of length 140 to max count: 140 [11:10:38.145160] Padding local tensor of length 140 to max count: 140 [11:10:38.145655] Padding local tensor of length 140 to max count: 140 [11:10:38.145952] Gathering scene names from all processes... [11:10:38.146089] Padding local tensor of length 140 on rank 0 to max count: 140 [11:10:38.147146] Evaluation metrics: [11:10:38.147337] psnr: 19.0453 [11:10:38.147436] ssim: 0.5260 [11:10:38.147520] lpips: 0.4533 [11:10:38.151320] Found 140 sequences [11:11:02.070710] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [11:11:02.959939] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [11:11:58.430583] Gathering scores from all processes... [11:11:58.431405] Padding local tensor of length 140 to max count: 140 [11:11:58.432077] Padding local tensor of length 140 to max count: 140 [11:11:58.432585] Padding local tensor of length 140 to max count: 140 [11:11:58.432880] Gathering scene names from all processes... [11:11:58.433062] Padding local tensor of length 140 on rank 0 to max count: 140 [11:11:58.434090] Evaluation metrics: [11:11:58.434294] psnr: 20.0202 [11:11:58.434398] ssim: 0.5558 [11:11:58.434501] lpips: 0.4207 [11:11:58.436151] [Eval][iter 25000] dl3dv_2view | psnr: 16.858255 | ssim: 0.464270 | lpips: 0.533192 || dl3dv_4view | psnr: 19.045271 | ssim: 0.526044 | lpips: 0.453275 || dl3dv_6view | psnr: 20.020243 | ssim: 0.555756 | lpips: 0.420685 [11:11:58.438890] [Eval][iter 25000] dl3dv_2view | psnr: 16.858255 | ssim: 0.464270 | lpips: 0.533192 || dl3dv_4view | psnr: 19.045271 | ssim: 0.526044 | lpips: 0.453275 || dl3dv_6view | psnr: 20.020243 | ssim: 0.555756 | lpips: 0.420685 [11:12:03.098276] Saved checkpoint at iteration 25000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [11:12:03.098622] Saved latest checkpoint at iteration 25000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [11:13:00.559569] [Epoch 1] | Iter: 25050 | Optim: 25050 | Time: 1.07s | LR: 0.000288 | SkipOpt: 0 loss: 0.193491 | l2_loss: 0.013806 | psnr: 18.599342 | perceptual_loss: 0.359370 | norm_perceptual_loss: 26.030132 [11:14:07.237227] [Epoch 1] | Iter: 25100 | Optim: 25100 | Time: 1.91s | LR: 0.000287 | SkipOpt: 0 loss: 0.210656 | l2_loss: 0.016998 | psnr: 17.695927 | perceptual_loss: 0.387315 | norm_perceptual_loss: 22.785418 [11:15:14.859915] [Epoch 1] | Iter: 25150 | Optim: 25150 | Time: 0.75s | LR: 0.000287 | SkipOpt: 0 loss: 0.223339 | l2_loss: 0.014876 | psnr: 18.275003 | perceptual_loss: 0.416925 | norm_perceptual_loss: 28.025826 [11:16:18.540721] [Epoch 1] | Iter: 25200 | Optim: 25200 | Time: 1.07s | LR: 0.000286 | SkipOpt: 0 loss: 0.190719 | l2_loss: 0.011382 | psnr: 19.437880 | perceptual_loss: 0.358674 | norm_perceptual_loss: 31.512894 [11:17:22.276311] [Epoch 1] | Iter: 25250 | Optim: 25250 | Time: 0.75s | LR: 0.000286 | SkipOpt: 0 loss: 0.154517 | l2_loss: 0.007003 | psnr: 21.547153 | perceptual_loss: 0.295027 | norm_perceptual_loss: 42.128624 [11:18:20.180542] [Epoch 1] | Iter: 25300 | Optim: 25300 | Time: 0.75s | LR: 0.000285 | SkipOpt: 0 loss: 0.208751 | l2_loss: 0.013889 | psnr: 18.573248 | perceptual_loss: 0.389723 | norm_perceptual_loss: 28.059544 [11:19:21.700113] [Epoch 1] | Iter: 25350 | Optim: 25350 | Time: 1.07s | LR: 0.000285 | SkipOpt: 0 loss: 0.206467 | l2_loss: 0.013796 | psnr: 18.602394 | perceptual_loss: 0.385342 | norm_perceptual_loss: 27.930973 [11:20:28.917249] [Epoch 1] | Iter: 25400 | Optim: 25400 | Time: 1.26s | LR: 0.000284 | SkipOpt: 0 loss: 0.202482 | l2_loss: 0.013884 | psnr: 18.574860 | perceptual_loss: 0.377196 | norm_perceptual_loss: 27.167740 [11:21:30.090480] [Epoch 1] | Iter: 25450 | Optim: 25450 | Time: 1.27s | LR: 0.000284 | SkipOpt: 0 loss: 0.240926 | l2_loss: 0.019355 | psnr: 17.132109 | perceptual_loss: 0.443142 | norm_perceptual_loss: 22.895678 [11:22:38.159422] [Epoch 1] | Iter: 25500 | Optim: 25500 | Time: 1.08s | LR: 0.000284 | SkipOpt: 0 loss: 0.197002 | l2_loss: 0.011714 | psnr: 19.313101 | perceptual_loss: 0.370576 | norm_perceptual_loss: 31.636419 [11:23:39.539869] [Epoch 1] | Iter: 25550 | Optim: 25550 | Time: 1.91s | LR: 0.000283 | SkipOpt: 0 loss: 0.230424 | l2_loss: 0.018790 | psnr: 17.260729 | perceptual_loss: 0.423267 | norm_perceptual_loss: 22.526186 [11:24:40.026649] [Epoch 1] | Iter: 25600 | Optim: 25600 | Time: 1.07s | LR: 0.000283 | SkipOpt: 0 loss: 0.183156 | l2_loss: 0.011699 | psnr: 19.318531 | perceptual_loss: 0.342914 | norm_perceptual_loss: 29.311548 [11:25:41.014509] [Epoch 1] | Iter: 25650 | Optim: 25650 | Time: 0.75s | LR: 0.000282 | SkipOpt: 0 loss: 0.180939 | l2_loss: 0.009596 | psnr: 20.178938 | perceptual_loss: 0.342686 | norm_perceptual_loss: 35.710056 [11:26:33.068106] [Epoch 1] | Iter: 25700 | Optim: 25700 | Time: 1.07s | LR: 0.000282 | SkipOpt: 0 loss: 0.227833 | l2_loss: 0.015274 | psnr: 18.160582 | perceptual_loss: 0.425118 | norm_perceptual_loss: 27.833511 [11:27:01.765685] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [11:27:34.703380] [Epoch 1] | Iter: 25750 | Optim: 25750 | Time: 1.08s | LR: 0.000281 | SkipOpt: 0 loss: 0.195512 | l2_loss: 0.011680 | psnr: 19.325676 | perceptual_loss: 0.367665 | norm_perceptual_loss: 31.478895 [11:28:36.848826] [Epoch 1] | Iter: 25800 | Optim: 25800 | Time: 1.07s | LR: 0.000281 | SkipOpt: 0 loss: 0.236166 | l2_loss: 0.015827 | psnr: 18.006084 | perceptual_loss: 0.440678 | norm_perceptual_loss: 27.843906 [11:29:41.219701] [Epoch 1] | Iter: 25850 | Optim: 25850 | Time: 1.91s | LR: 0.000280 | SkipOpt: 0 loss: 0.213056 | l2_loss: 0.016008 | psnr: 17.956551 | perceptual_loss: 0.394095 | norm_perceptual_loss: 24.618187 [11:30:34.353982] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [11:30:39.357829] [Epoch 1] | Iter: 25900 | Optim: 25900 | Time: 0.75s | LR: 0.000280 | SkipOpt: 0 loss: 0.204933 | l2_loss: 0.015205 | psnr: 18.180073 | perceptual_loss: 0.379456 | norm_perceptual_loss: 24.955626 [11:31:39.562366] [Epoch 1] | Iter: 25950 | Optim: 25950 | Time: 1.90s | LR: 0.000279 | SkipOpt: 0 loss: 0.223128 | l2_loss: 0.017739 | psnr: 17.510798 | perceptual_loss: 0.410779 | norm_perceptual_loss: 23.157297 [11:32:42.165328] [Epoch 1] | Iter: 26000 | Optim: 26000 | Time: 1.91s | LR: 0.000279 | SkipOpt: 0 loss: 0.232973 | l2_loss: 0.021596 | psnr: 16.656214 | perceptual_loss: 0.422754 | norm_perceptual_loss: 19.575310 [11:32:47.717197] Saved checkpoint at iteration 26000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [11:32:47.717570] Saved latest checkpoint at iteration 26000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [11:33:39.738355] [Epoch 1] | Iter: 26050 | Optim: 26050 | Time: 0.75s | LR: 0.000278 | SkipOpt: 0 loss: 0.222577 | l2_loss: 0.014937 | psnr: 18.257378 | perceptual_loss: 0.415281 | norm_perceptual_loss: 27.802212 [11:34:42.615364] [Epoch 1] | Iter: 26100 | Optim: 26100 | Time: 0.75s | LR: 0.000278 | SkipOpt: 0 loss: 0.203064 | l2_loss: 0.009797 | psnr: 20.088928 | perceptual_loss: 0.386534 | norm_perceptual_loss: 39.453060 [11:35:45.372195] [Epoch 1] | Iter: 26150 | Optim: 26150 | Time: 1.07s | LR: 0.000277 | SkipOpt: 0 loss: 0.223465 | l2_loss: 0.013189 | psnr: 18.797834 | perceptual_loss: 0.420552 | norm_perceptual_loss: 31.886248 [11:36:52.755074] [Epoch 1] | Iter: 26200 | Optim: 26200 | Time: 1.27s | LR: 0.000277 | SkipOpt: 0 loss: 0.211455 | l2_loss: 0.013402 | psnr: 18.728464 | perceptual_loss: 0.396107 | norm_perceptual_loss: 29.556911 [11:37:48.872829] [Epoch 1] | Iter: 26250 | Optim: 26250 | Time: 1.07s | LR: 0.000277 | SkipOpt: 0 loss: 0.184666 | l2_loss: 0.010920 | psnr: 19.617861 | perceptual_loss: 0.347492 | norm_perceptual_loss: 31.822247 [11:38:54.110208] [Epoch 1] | Iter: 26300 | Optim: 26300 | Time: 1.08s | LR: 0.000276 | SkipOpt: 0 loss: 0.216262 | l2_loss: 0.013091 | psnr: 18.830393 | perceptual_loss: 0.406342 | norm_perceptual_loss: 31.040668 [11:39:52.692956] [Epoch 1] | Iter: 26350 | Optim: 26350 | Time: 1.91s | LR: 0.000276 | SkipOpt: 0 loss: 0.256561 | l2_loss: 0.020507 | psnr: 16.880968 | perceptual_loss: 0.472109 | norm_perceptual_loss: 23.021786 [11:40:51.133035] [Epoch 1] | Iter: 26400 | Optim: 26400 | Time: 1.07s | LR: 0.000275 | SkipOpt: 0 loss: 0.180034 | l2_loss: 0.009951 | psnr: 20.021456 | perceptual_loss: 0.340166 | norm_perceptual_loss: 34.185078 [11:41:43.440115] [Epoch 1] | Iter: 26450 | Optim: 26450 | Time: 1.07s | LR: 0.000275 | SkipOpt: 0 loss: 0.252222 | l2_loss: 0.021467 | psnr: 16.682331 | perceptual_loss: 0.461510 | norm_perceptual_loss: 21.498817 [11:42:45.240262] [Epoch 1] | Iter: 26500 | Optim: 26500 | Time: 0.75s | LR: 0.000274 | SkipOpt: 0 loss: 0.222184 | l2_loss: 0.014280 | psnr: 18.452719 | perceptual_loss: 0.415808 | norm_perceptual_loss: 29.118221 [11:43:44.221759] [Epoch 1] | Iter: 26550 | Optim: 26550 | Time: 1.26s | LR: 0.000274 | SkipOpt: 0 loss: 0.240951 | l2_loss: 0.023582 | psnr: 16.274267 | perceptual_loss: 0.434738 | norm_perceptual_loss: 18.435478 [11:44:47.746609] [Epoch 1] | Iter: 26600 | Optim: 26600 | Time: 1.26s | LR: 0.000273 | SkipOpt: 0 loss: 0.257605 | l2_loss: 0.020754 | psnr: 16.829044 | perceptual_loss: 0.473703 | norm_perceptual_loss: 22.824991 [11:45:53.115149] [Epoch 1] | Iter: 26650 | Optim: 26650 | Time: 1.07s | LR: 0.000273 | SkipOpt: 0 loss: 0.202222 | l2_loss: 0.013672 | psnr: 18.641714 | perceptual_loss: 0.377101 | norm_perceptual_loss: 27.582207 [11:46:53.512058] [Epoch 1] | Iter: 26700 | Optim: 26700 | Time: 1.26s | LR: 0.000272 | SkipOpt: 0 loss: 0.196133 | l2_loss: 0.011600 | psnr: 19.355600 | perceptual_loss: 0.369066 | norm_perceptual_loss: 31.817371 [11:47:56.299139] [Epoch 1] | Iter: 26750 | Optim: 26750 | Time: 1.90s | LR: 0.000272 | SkipOpt: 0 loss: 0.223482 | l2_loss: 0.016245 | psnr: 17.892693 | perceptual_loss: 0.414474 | norm_perceptual_loss: 25.513277 [11:48:59.800799] [Epoch 1] | Iter: 26800 | Optim: 26800 | Time: 1.90s | LR: 0.000271 | SkipOpt: 0 loss: 0.202831 | l2_loss: 0.015876 | psnr: 17.992460 | perceptual_loss: 0.373910 | norm_perceptual_loss: 23.551174 [11:49:55.138070] [Epoch 1] | Iter: 26850 | Optim: 26850 | Time: 1.26s | LR: 0.000271 | SkipOpt: 0 loss: 0.210033 | l2_loss: 0.018003 | psnr: 17.446499 | perceptual_loss: 0.384060 | norm_perceptual_loss: 21.332863 [11:50:56.463324] [Epoch 1] | Iter: 26900 | Optim: 26900 | Time: 0.75s | LR: 0.000270 | SkipOpt: 0 loss: 0.175776 | l2_loss: 0.009450 | psnr: 20.245512 | perceptual_loss: 0.332651 | norm_perceptual_loss: 35.199768 [11:51:56.362438] [Epoch 1] | Iter: 26950 | Optim: 26950 | Time: 1.91s | LR: 0.000270 | SkipOpt: 0 loss: 0.237229 | l2_loss: 0.016123 | psnr: 17.925484 | perceptual_loss: 0.442211 | norm_perceptual_loss: 27.426958 [11:53:03.072255] [Epoch 1] | Iter: 27000 | Optim: 27000 | Time: 0.75s | LR: 0.000269 | SkipOpt: 0 loss: 0.224933 | l2_loss: 0.013867 | psnr: 18.580097 | perceptual_loss: 0.422132 | norm_perceptual_loss: 30.440912 [11:53:09.177047] Saved checkpoint at iteration 27000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [11:53:09.177695] Saved latest checkpoint at iteration 27000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [11:54:09.142871] [Epoch 1] | Iter: 27050 | Optim: 27050 | Time: 1.07s | LR: 0.000269 | SkipOpt: 0 loss: 0.171970 | l2_loss: 0.008851 | psnr: 20.529900 | perceptual_loss: 0.326238 | norm_perceptual_loss: 36.857368 [11:55:15.537001] [Epoch 1] | Iter: 27100 | Optim: 27100 | Time: 1.91s | LR: 0.000269 | SkipOpt: 0 loss: 0.251381 | l2_loss: 0.022491 | psnr: 16.479872 | perceptual_loss: 0.457780 | norm_perceptual_loss: 20.353735 [11:56:10.641343] [Epoch 1] | Iter: 27150 | Optim: 27150 | Time: 0.85s | LR: 0.000268 | SkipOpt: 0 loss: 0.191648 | l2_loss: 0.012750 | psnr: 18.944942 | perceptual_loss: 0.357796 | norm_perceptual_loss: 28.062683 [11:57:11.956500] [Epoch 1] | Iter: 27200 | Optim: 27200 | Time: 1.91s | LR: 0.000268 | SkipOpt: 0 loss: 0.223787 | l2_loss: 0.017372 | psnr: 17.601572 | perceptual_loss: 0.412831 | norm_perceptual_loss: 23.764521 [11:58:15.387968] [Epoch 1] | Iter: 27250 | Optim: 27250 | Time: 0.75s | LR: 0.000267 | SkipOpt: 0 loss: 0.189045 | l2_loss: 0.007762 | psnr: 21.100185 | perceptual_loss: 0.362566 | norm_perceptual_loss: 46.709515 [11:59:19.462030] [Epoch 1] | Iter: 27300 | Optim: 27300 | Time: 1.90s | LR: 0.000267 | SkipOpt: 0 loss: 0.206225 | l2_loss: 0.016024 | psnr: 17.952309 | perceptual_loss: 0.380402 | norm_perceptual_loss: 23.739635 [12:00:21.518305] [Epoch 1] | Iter: 27350 | Optim: 27350 | Time: 1.07s | LR: 0.000266 | SkipOpt: 0 loss: 0.199593 | l2_loss: 0.010501 | psnr: 19.787828 | perceptual_loss: 0.378185 | norm_perceptual_loss: 36.015289 [12:01:32.264408] [Epoch 1] | Iter: 27400 | Optim: 27400 | Time: 1.07s | LR: 0.000266 | SkipOpt: 0 loss: 0.231630 | l2_loss: 0.015564 | psnr: 18.078798 | perceptual_loss: 0.432131 | norm_perceptual_loss: 27.764868 [12:02:32.407226] [Epoch 1] | Iter: 27450 | Optim: 27450 | Time: 1.07s | LR: 0.000265 | SkipOpt: 0 loss: 0.215630 | l2_loss: 0.013292 | psnr: 18.764194 | perceptual_loss: 0.404676 | norm_perceptual_loss: 30.445734 [12:03:37.286430] [Epoch 1] | Iter: 27500 | Optim: 27500 | Time: 1.08s | LR: 0.000265 | SkipOpt: 0 loss: 0.189898 | l2_loss: 0.013012 | psnr: 18.856632 | perceptual_loss: 0.353772 | norm_perceptual_loss: 27.188581 [12:04:39.874872] [Epoch 1] | Iter: 27550 | Optim: 27550 | Time: 1.90s | LR: 0.000264 | SkipOpt: 0 loss: 0.201585 | l2_loss: 0.016511 | psnr: 17.822205 | perceptual_loss: 0.370148 | norm_perceptual_loss: 22.417948 [12:05:47.438069] [Epoch 1] | Iter: 27600 | Optim: 27600 | Time: 1.91s | LR: 0.000264 | SkipOpt: 0 loss: 0.215172 | l2_loss: 0.014520 | psnr: 18.380270 | perceptual_loss: 0.401304 | norm_perceptual_loss: 27.637590 [12:06:51.957691] [Epoch 1] | Iter: 27650 | Optim: 27650 | Time: 0.76s | LR: 0.000263 | SkipOpt: 0 loss: 0.166118 | l2_loss: 0.007205 | psnr: 21.423426 | perceptual_loss: 0.317824 | norm_perceptual_loss: 44.109253 [12:07:55.657276] [Epoch 1] | Iter: 27700 | Optim: 27700 | Time: 1.91s | LR: 0.000263 | SkipOpt: 0 loss: 0.238185 | l2_loss: 0.017824 | psnr: 17.490007 | perceptual_loss: 0.440722 | norm_perceptual_loss: 24.726669 [12:08:58.589185] [Epoch 1] | Iter: 27750 | Optim: 27750 | Time: 1.08s | LR: 0.000262 | SkipOpt: 0 loss: 0.175812 | l2_loss: 0.010779 | psnr: 19.674385 | perceptual_loss: 0.330067 | norm_perceptual_loss: 30.622492 [12:09:57.343568] [Epoch 1] | Iter: 27800 | Optim: 27800 | Time: 0.85s | LR: 0.000262 | SkipOpt: 0 loss: 0.223976 | l2_loss: 0.014004 | psnr: 18.537560 | perceptual_loss: 0.419944 | norm_perceptual_loss: 29.988028 [12:10:52.146751] [Epoch 1] | Iter: 27850 | Optim: 27850 | Time: 1.92s | LR: 0.000261 | SkipOpt: 0 loss: 0.240289 | l2_loss: 0.020224 | psnr: 16.941298 | perceptual_loss: 0.440129 | norm_perceptual_loss: 21.762541 [12:11:55.717818] [Epoch 1] | Iter: 27900 | Optim: 27900 | Time: 0.75s | LR: 0.000261 | SkipOpt: 0 loss: 0.204963 | l2_loss: 0.013578 | psnr: 18.671669 | perceptual_loss: 0.382770 | norm_perceptual_loss: 28.190659 [12:12:52.100092] [Epoch 1] | Iter: 27950 | Optim: 27950 | Time: 1.07s | LR: 0.000260 | SkipOpt: 0 loss: 0.188767 | l2_loss: 0.008852 | psnr: 20.529776 | perceptual_loss: 0.359830 | norm_perceptual_loss: 40.651321 [12:13:57.102683] [Epoch 1] | Iter: 28000 | Optim: 28000 | Time: 0.75s | LR: 0.000260 | SkipOpt: 0 loss: 0.187043 | l2_loss: 0.011764 | psnr: 19.294432 | perceptual_loss: 0.350558 | norm_perceptual_loss: 29.799091 [12:14:02.732135] Saved checkpoint at iteration 28000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [12:14:02.732519] Saved latest checkpoint at iteration 28000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [12:15:03.384187] [Epoch 1] | Iter: 28050 | Optim: 28050 | Time: 1.91s | LR: 0.000259 | SkipOpt: 0 loss: 0.190885 | l2_loss: 0.013572 | psnr: 18.673708 | perceptual_loss: 0.354627 | norm_perceptual_loss: 26.130209 [12:16:10.524621] [Epoch 1] | Iter: 28100 | Optim: 28100 | Time: 1.91s | LR: 0.000259 | SkipOpt: 0 loss: 0.249579 | l2_loss: 0.019510 | psnr: 17.097412 | perceptual_loss: 0.460138 | norm_perceptual_loss: 23.584658 [12:17:17.191078] [Epoch 1] | Iter: 28150 | Optim: 28150 | Time: 1.08s | LR: 0.000258 | SkipOpt: 0 loss: 0.195351 | l2_loss: 0.010054 | psnr: 19.976421 | perceptual_loss: 0.370594 | norm_perceptual_loss: 36.858723 [12:18:22.123357] [Epoch 1] | Iter: 28200 | Optim: 28200 | Time: 1.92s | LR: 0.000258 | SkipOpt: 0 loss: 0.249996 | l2_loss: 0.021162 | psnr: 16.744389 | perceptual_loss: 0.457668 | norm_perceptual_loss: 21.626640 [12:19:22.253168] [Epoch 1] | Iter: 28250 | Optim: 28250 | Time: 1.26s | LR: 0.000257 | SkipOpt: 0 loss: 0.206627 | l2_loss: 0.014202 | psnr: 18.476450 | perceptual_loss: 0.384849 | norm_perceptual_loss: 27.097891 [12:20:26.156904] [Epoch 1] | Iter: 28300 | Optim: 28300 | Time: 1.91s | LR: 0.000257 | SkipOpt: 0 loss: 0.214925 | l2_loss: 0.016247 | psnr: 17.892141 | perceptual_loss: 0.397354 | norm_perceptual_loss: 24.456379 [12:21:19.202550] [Epoch 1] | Iter: 28350 | Optim: 28350 | Time: 0.75s | LR: 0.000256 | SkipOpt: 0 loss: 0.196205 | l2_loss: 0.009344 | psnr: 20.294769 | perceptual_loss: 0.373723 | norm_perceptual_loss: 39.996964 [12:22:20.857807] [Epoch 1] | Iter: 28400 | Optim: 28400 | Time: 1.91s | LR: 0.000256 | SkipOpt: 0 loss: 0.232618 | l2_loss: 0.017881 | psnr: 17.476004 | perceptual_loss: 0.429472 | norm_perceptual_loss: 24.017944 [12:23:27.904596] [Epoch 1] | Iter: 28450 | Optim: 28450 | Time: 1.91s | LR: 0.000256 | SkipOpt: 0 loss: 0.217774 | l2_loss: 0.016950 | psnr: 17.708185 | perceptual_loss: 0.401646 | norm_perceptual_loss: 23.695303 [12:24:24.033864] [Epoch 1] | Iter: 28500 | Optim: 28500 | Time: 0.75s | LR: 0.000255 | SkipOpt: 0 loss: 0.170594 | l2_loss: 0.009439 | psnr: 20.250809 | perceptual_loss: 0.322310 | norm_perceptual_loss: 34.147129 [12:25:24.124306] [Epoch 1] | Iter: 28550 | Optim: 28550 | Time: 1.27s | LR: 0.000255 | SkipOpt: 0 loss: 0.194221 | l2_loss: 0.011987 | psnr: 19.212950 | perceptual_loss: 0.364467 | norm_perceptual_loss: 30.405598 [12:26:28.307934] [Epoch 1] | Iter: 28600 | Optim: 28600 | Time: 1.91s | LR: 0.000254 | SkipOpt: 0 loss: 0.190276 | l2_loss: 0.011926 | psnr: 19.235123 | perceptual_loss: 0.356699 | norm_perceptual_loss: 29.909874 [12:27:31.876720] [Epoch 1] | Iter: 28650 | Optim: 28650 | Time: 0.75s | LR: 0.000254 | SkipOpt: 0 loss: 0.182322 | l2_loss: 0.011454 | psnr: 19.410404 | perceptual_loss: 0.341736 | norm_perceptual_loss: 29.835384 [12:28:25.983495] [Epoch 1] | Iter: 28700 | Optim: 28700 | Time: 1.07s | LR: 0.000253 | SkipOpt: 0 loss: 0.190404 | l2_loss: 0.012800 | psnr: 18.927977 | perceptual_loss: 0.355208 | norm_perceptual_loss: 27.751062 [12:29:29.831824] [Epoch 1] | Iter: 28750 | Optim: 28750 | Time: 1.91s | LR: 0.000253 | SkipOpt: 0 loss: 0.211673 | l2_loss: 0.017483 | psnr: 17.573938 | perceptual_loss: 0.388380 | norm_perceptual_loss: 22.215231 [12:30:31.390596] [Epoch 1] | Iter: 28800 | Optim: 28800 | Time: 1.91s | LR: 0.000252 | SkipOpt: 0 loss: 0.233119 | l2_loss: 0.020062 | psnr: 16.976173 | perceptual_loss: 0.426113 | norm_perceptual_loss: 21.239384 [12:31:32.686633] [Epoch 1] | Iter: 28850 | Optim: 28850 | Time: 1.91s | LR: 0.000252 | SkipOpt: 0 loss: 0.188683 | l2_loss: 0.013269 | psnr: 18.771526 | perceptual_loss: 0.350827 | norm_perceptual_loss: 26.439060 [12:32:38.211708] [Epoch 1] | Iter: 28900 | Optim: 28900 | Time: 0.85s | LR: 0.000251 | SkipOpt: 0 loss: 0.195294 | l2_loss: 0.011672 | psnr: 19.328432 | perceptual_loss: 0.367243 | norm_perceptual_loss: 31.462778 [12:33:40.699208] [Epoch 1] | Iter: 28950 | Optim: 28950 | Time: 1.90s | LR: 0.000251 | SkipOpt: 0 loss: 0.236718 | l2_loss: 0.021127 | psnr: 16.751699 | perceptual_loss: 0.431182 | norm_perceptual_loss: 20.409416 [12:34:39.927425] [Epoch 1] | Iter: 29000 | Optim: 29000 | Time: 1.08s | LR: 0.000250 | SkipOpt: 0 loss: 0.206876 | l2_loss: 0.019770 | psnr: 17.039873 | perceptual_loss: 0.374211 | norm_perceptual_loss: 18.927958 [12:34:46.218582] Saved checkpoint at iteration 29000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [12:34:46.218937] Saved latest checkpoint at iteration 29000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [12:35:51.636006] [Epoch 1] | Iter: 29050 | Optim: 29050 | Time: 0.76s | LR: 0.000250 | SkipOpt: 0 loss: 0.170265 | l2_loss: 0.010131 | psnr: 19.943514 | perceptual_loss: 0.320269 | norm_perceptual_loss: 31.613026 [12:36:57.713909] [Epoch 1] | Iter: 29100 | Optim: 29100 | Time: 1.07s | LR: 0.000249 | SkipOpt: 0 loss: 0.204232 | l2_loss: 0.014903 | psnr: 18.267157 | perceptual_loss: 0.378657 | norm_perceptual_loss: 25.407459 [12:37:54.291649] [Epoch 1] | Iter: 29150 | Optim: 29150 | Time: 1.91s | LR: 0.000249 | SkipOpt: 0 loss: 0.204084 | l2_loss: 0.015198 | psnr: 18.182201 | perceptual_loss: 0.377771 | norm_perceptual_loss: 24.857025 [12:38:56.150924] [Epoch 1] | Iter: 29200 | Optim: 29200 | Time: 1.07s | LR: 0.000248 | SkipOpt: 0 loss: 0.225407 | l2_loss: 0.018571 | psnr: 17.311686 | perceptual_loss: 0.413673 | norm_perceptual_loss: 22.275419 [12:39:58.588557] [Epoch 1] | Iter: 29250 | Optim: 29250 | Time: 1.91s | LR: 0.000248 | SkipOpt: 0 loss: 0.237368 | l2_loss: 0.019402 | psnr: 17.121428 | perceptual_loss: 0.435931 | norm_perceptual_loss: 22.467781 [12:41:03.041398] [Epoch 1] | Iter: 29300 | Optim: 29300 | Time: 1.91s | LR: 0.000247 | SkipOpt: 0 loss: 0.213463 | l2_loss: 0.015091 | psnr: 18.212822 | perceptual_loss: 0.396744 | norm_perceptual_loss: 26.290133 [12:41:57.108960] [Epoch 1] | Iter: 29350 | Optim: 29350 | Time: 0.75s | LR: 0.000247 | SkipOpt: 0 loss: 0.220192 | l2_loss: 0.015904 | psnr: 17.985022 | perceptual_loss: 0.408576 | norm_perceptual_loss: 25.690639 [12:42:55.106085] [Epoch 1] | Iter: 29400 | Optim: 29400 | Time: 0.75s | LR: 0.000246 | SkipOpt: 0 loss: 0.164941 | l2_loss: 0.007864 | psnr: 21.043434 | perceptual_loss: 0.314153 | norm_perceptual_loss: 39.946991 [12:44:01.436104] [Epoch 1] | Iter: 29450 | Optim: 29450 | Time: 1.07s | LR: 0.000246 | SkipOpt: 0 loss: 0.184741 | l2_loss: 0.009738 | psnr: 20.115499 | perceptual_loss: 0.350007 | norm_perceptual_loss: 35.944016 [12:45:01.318318] [Epoch 1] | Iter: 29500 | Optim: 29500 | Time: 0.75s | LR: 0.000245 | SkipOpt: 0 loss: 0.217082 | l2_loss: 0.014290 | psnr: 18.449581 | perceptual_loss: 0.405584 | norm_perceptual_loss: 28.381731 [12:46:08.304597] [Epoch 1] | Iter: 29550 | Optim: 29550 | Time: 0.75s | LR: 0.000245 | SkipOpt: 0 loss: 0.192446 | l2_loss: 0.010100 | psnr: 19.956764 | perceptual_loss: 0.364692 | norm_perceptual_loss: 36.107922 [12:47:09.836236] [Epoch 1] | Iter: 29600 | Optim: 29600 | Time: 1.08s | LR: 0.000244 | SkipOpt: 0 loss: 0.218729 | l2_loss: 0.014766 | psnr: 18.307241 | perceptual_loss: 0.407925 | norm_perceptual_loss: 27.625149 [12:48:07.389598] [Epoch 1] | Iter: 29650 | Optim: 29650 | Time: 1.91s | LR: 0.000244 | SkipOpt: 0 loss: 0.221344 | l2_loss: 0.015676 | psnr: 18.047531 | perceptual_loss: 0.411335 | norm_perceptual_loss: 26.239120 [12:49:10.832418] [Epoch 1] | Iter: 29700 | Optim: 29700 | Time: 1.08s | LR: 0.000243 | SkipOpt: 0 loss: 0.204649 | l2_loss: 0.013460 | psnr: 18.709463 | perceptual_loss: 0.382378 | norm_perceptual_loss: 28.407906 [12:50:12.639333] [Epoch 1] | Iter: 29750 | Optim: 29750 | Time: 1.07s | LR: 0.000243 | SkipOpt: 0 loss: 0.192494 | l2_loss: 0.012878 | psnr: 18.901468 | perceptual_loss: 0.359231 | norm_perceptual_loss: 27.894629 [12:51:08.824051] [Epoch 1] | Iter: 29800 | Optim: 29800 | Time: 1.07s | LR: 0.000242 | SkipOpt: 0 loss: 0.206380 | l2_loss: 0.012847 | psnr: 18.912066 | perceptual_loss: 0.387066 | norm_perceptual_loss: 30.129438 [12:52:08.484745] [Epoch 1] | Iter: 29850 | Optim: 29850 | Time: 0.75s | LR: 0.000242 | SkipOpt: 0 loss: 0.178817 | l2_loss: 0.010308 | psnr: 19.868147 | perceptual_loss: 0.337017 | norm_perceptual_loss: 32.693913 [12:53:08.630881] [Epoch 1] | Iter: 29900 | Optim: 29900 | Time: 1.91s | LR: 0.000241 | SkipOpt: 0 loss: 0.234157 | l2_loss: 0.019260 | psnr: 17.153490 | perceptual_loss: 0.429795 | norm_perceptual_loss: 22.315718 [12:54:10.624928] [Epoch 1] | Iter: 29950 | Optim: 29950 | Time: 1.07s | LR: 0.000241 | SkipOpt: 0 loss: 0.187162 | l2_loss: 0.012307 | psnr: 19.098412 | perceptual_loss: 0.349710 | norm_perceptual_loss: 28.415064 [12:55:05.465629] [Epoch 1] | Iter: 30000 | Optim: 30000 | Time: 0.75s | LR: 0.000240 | SkipOpt: 0 loss: 0.179063 | l2_loss: 0.009266 | psnr: 20.330990 | perceptual_loss: 0.339593 | norm_perceptual_loss: 36.648624 [12:55:07.201846] Found 140 sequences [12:55:22.422384] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [12:55:23.279009] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [12:56:18.798489] Gathering scores from all processes... [12:56:18.799285] Padding local tensor of length 140 to max count: 140 [12:56:18.799818] Padding local tensor of length 140 to max count: 140 [12:56:18.800324] Padding local tensor of length 140 to max count: 140 [12:56:18.800583] Gathering scene names from all processes... [12:56:18.800692] Padding local tensor of length 140 on rank 0 to max count: 140 [12:56:18.801678] Evaluation metrics: [12:56:18.801847] psnr: 16.9589 [12:56:18.801934] ssim: 0.4699 [12:56:18.802009] lpips: 0.5213 [12:56:18.805407] Found 140 sequences [12:56:42.719655] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [12:56:43.566553] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [12:57:36.145876] Gathering scores from all processes... [12:57:36.146627] Padding local tensor of length 140 to max count: 140 [12:57:36.147181] Padding local tensor of length 140 to max count: 140 [12:57:36.147617] Padding local tensor of length 140 to max count: 140 [12:57:36.147869] Gathering scene names from all processes... [12:57:36.147991] Padding local tensor of length 140 on rank 0 to max count: 140 [12:57:36.149005] Evaluation metrics: [12:57:36.149179] psnr: 19.3424 [12:57:36.149262] ssim: 0.5421 [12:57:36.149336] lpips: 0.4309 [12:57:36.152798] Found 140 sequences [12:57:59.346199] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [12:58:00.196046] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [12:58:55.744632] Gathering scores from all processes... [12:58:55.745432] Padding local tensor of length 140 to max count: 140 [12:58:55.746050] Padding local tensor of length 140 to max count: 140 [12:58:55.746589] Padding local tensor of length 140 to max count: 140 [12:58:55.746887] Gathering scene names from all processes... [12:58:55.747027] Padding local tensor of length 140 on rank 0 to max count: 140 [12:58:55.747964] Evaluation metrics: [12:58:55.748167] psnr: 20.3105 [12:58:55.748256] ssim: 0.5734 [12:58:55.748356] lpips: 0.3967 [12:58:55.749826] [Eval][iter 30000] dl3dv_2view | psnr: 16.958929 | ssim: 0.469906 | lpips: 0.521318 || dl3dv_4view | psnr: 19.342415 | ssim: 0.542110 | lpips: 0.430941 || dl3dv_6view | psnr: 20.310455 | ssim: 0.573381 | lpips: 0.396702 [12:58:55.752607] [Eval][iter 30000] dl3dv_2view | psnr: 16.958929 | ssim: 0.469906 | lpips: 0.521318 || dl3dv_4view | psnr: 19.342415 | ssim: 0.542110 | lpips: 0.430941 || dl3dv_6view | psnr: 20.310455 | ssim: 0.573381 | lpips: 0.396702 [12:59:00.346603] Saved checkpoint at iteration 30000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [12:59:00.346964] Saved latest checkpoint at iteration 30000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [13:00:01.083182] [Epoch 1] | Iter: 30050 | Optim: 30050 | Time: 0.85s | LR: 0.000240 | SkipOpt: 0 loss: 0.186307 | l2_loss: 0.009768 | psnr: 20.102139 | perceptual_loss: 0.353078 | norm_perceptual_loss: 36.148022 [13:01:06.815354] [Epoch 1] | Iter: 30100 | Optim: 30100 | Time: 0.75s | LR: 0.000239 | SkipOpt: 0 loss: 0.194788 | l2_loss: 0.008269 | psnr: 20.825718 | perceptual_loss: 0.373040 | norm_perceptual_loss: 45.115597 [13:02:10.471365] [Epoch 1] | Iter: 30150 | Optim: 30150 | Time: 0.75s | LR: 0.000239 | SkipOpt: 0 loss: 0.178317 | l2_loss: 0.009730 | psnr: 20.118820 | perceptual_loss: 0.337173 | norm_perceptual_loss: 34.652546 [13:03:10.851217] [Epoch 1] | Iter: 30200 | Optim: 30200 | Time: 1.91s | LR: 0.000238 | SkipOpt: 0 loss: 0.207757 | l2_loss: 0.016106 | psnr: 17.930254 | perceptual_loss: 0.383304 | norm_perceptual_loss: 23.799530 [13:04:14.508754] [Epoch 1] | Iter: 30250 | Optim: 30250 | Time: 1.08s | LR: 0.000238 | SkipOpt: 0 loss: 0.216239 | l2_loss: 0.015403 | psnr: 18.123867 | perceptual_loss: 0.401672 | norm_perceptual_loss: 26.077057 [13:05:16.689197] [Epoch 1] | Iter: 30300 | Optim: 30300 | Time: 1.90s | LR: 0.000237 | SkipOpt: 0 loss: 0.263664 | l2_loss: 0.020369 | psnr: 16.910378 | perceptual_loss: 0.486590 | norm_perceptual_loss: 23.889172 [13:06:20.513935] [Epoch 1] | Iter: 30350 | Optim: 30350 | Time: 1.07s | LR: 0.000237 | SkipOpt: 0 loss: 0.200196 | l2_loss: 0.013018 | psnr: 18.854506 | perceptual_loss: 0.374356 | norm_perceptual_loss: 28.756445 [13:07:28.692163] [Epoch 1] | Iter: 30400 | Optim: 30400 | Time: 1.91s | LR: 0.000236 | SkipOpt: 0 loss: 0.227612 | l2_loss: 0.018418 | psnr: 17.347486 | perceptual_loss: 0.418388 | norm_perceptual_loss: 22.715773 [13:08:34.592888] [Epoch 1] | Iter: 30450 | Optim: 30450 | Time: 1.91s | LR: 0.000236 | SkipOpt: 0 loss: 0.204622 | l2_loss: 0.017826 | psnr: 17.489397 | perceptual_loss: 0.373592 | norm_perceptual_loss: 20.957403 [13:09:39.883169] [Epoch 1] | Iter: 30500 | Optim: 30500 | Time: 1.92s | LR: 0.000235 | SkipOpt: 0 loss: 0.203977 | l2_loss: 0.014804 | psnr: 18.296328 | perceptual_loss: 0.378347 | norm_perceptual_loss: 25.557756 [13:10:46.349375] [Epoch 1] | Iter: 30550 | Optim: 30550 | Time: 1.91s | LR: 0.000235 | SkipOpt: 0 loss: 0.215075 | l2_loss: 0.017366 | psnr: 17.603100 | perceptual_loss: 0.395419 | norm_perceptual_loss: 22.770231 [13:11:51.363782] [Epoch 1] | Iter: 30600 | Optim: 30600 | Time: 1.90s | LR: 0.000234 | SkipOpt: 0 loss: 0.215564 | l2_loss: 0.015726 | psnr: 18.033686 | perceptual_loss: 0.399675 | norm_perceptual_loss: 25.414164 [13:12:56.233881] [Epoch 1] | Iter: 30650 | Optim: 30650 | Time: 0.75s | LR: 0.000234 | SkipOpt: 0 loss: 0.183912 | l2_loss: 0.010357 | psnr: 19.847462 | perceptual_loss: 0.347108 | norm_perceptual_loss: 33.512817 [13:13:54.106617] [Epoch 1] | Iter: 30700 | Optim: 30700 | Time: 1.91s | LR: 0.000233 | SkipOpt: 0 loss: 0.234205 | l2_loss: 0.019423 | psnr: 17.116734 | perceptual_loss: 0.429562 | norm_perceptual_loss: 22.115629 [13:14:52.929218] [Epoch 1] | Iter: 30750 | Optim: 30750 | Time: 0.75s | LR: 0.000233 | SkipOpt: 0 loss: 0.177082 | l2_loss: 0.008264 | psnr: 20.828074 | perceptual_loss: 0.337635 | norm_perceptual_loss: 40.855938 [13:15:56.018764] [Epoch 1] | Iter: 30800 | Optim: 30800 | Time: 0.75s | LR: 0.000232 | SkipOpt: 0 loss: 0.121416 | l2_loss: 0.004520 | psnr: 23.448845 | perceptual_loss: 0.233792 | norm_perceptual_loss: 51.726658 [13:16:57.500915] [Epoch 1] | Iter: 30850 | Optim: 30850 | Time: 1.91s | LR: 0.000232 | SkipOpt: 0 loss: 0.182065 | l2_loss: 0.012133 | psnr: 19.160416 | perceptual_loss: 0.339864 | norm_perceptual_loss: 28.012188 [13:18:04.055899] [Epoch 1] | Iter: 30900 | Optim: 30900 | Time: 1.07s | LR: 0.000231 | SkipOpt: 0 loss: 0.196765 | l2_loss: 0.011547 | psnr: 19.375450 | perceptual_loss: 0.370437 | norm_perceptual_loss: 32.081860 [13:19:10.124550] [Epoch 1] | Iter: 30950 | Optim: 30950 | Time: 1.27s | LR: 0.000231 | SkipOpt: 0 loss: 0.217633 | l2_loss: 0.015708 | psnr: 18.038790 | perceptual_loss: 0.403850 | norm_perceptual_loss: 25.709845 [13:20:12.417007] [Epoch 1] | Iter: 31000 | Optim: 31000 | Time: 1.91s | LR: 0.000230 | SkipOpt: 0 loss: 0.219700 | l2_loss: 0.018742 | psnr: 17.271910 | perceptual_loss: 0.401918 | norm_perceptual_loss: 21.445089 [13:20:18.948585] Saved checkpoint at iteration 31000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [13:20:18.949215] Saved latest checkpoint at iteration 31000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [13:21:18.598298] [Epoch 1] | Iter: 31050 | Optim: 31050 | Time: 1.27s | LR: 0.000230 | SkipOpt: 0 loss: 0.180170 | l2_loss: 0.010449 | psnr: 19.809229 | perceptual_loss: 0.339441 | norm_perceptual_loss: 32.485321 [13:22:18.921540] [Epoch 1] | Iter: 31100 | Optim: 31100 | Time: 0.75s | LR: 0.000229 | SkipOpt: 0 loss: 0.148814 | l2_loss: 0.007753 | psnr: 21.105049 | perceptual_loss: 0.282122 | norm_perceptual_loss: 36.386578 [13:23:18.725700] [Epoch 1] | Iter: 31150 | Optim: 31150 | Time: 1.91s | LR: 0.000229 | SkipOpt: 0 loss: 0.221423 | l2_loss: 0.014606 | psnr: 18.354551 | perceptual_loss: 0.413634 | norm_perceptual_loss: 28.318541 [13:24:19.553442] [Epoch 1] | Iter: 31200 | Optim: 31200 | Time: 1.07s | LR: 0.000228 | SkipOpt: 0 loss: 0.222637 | l2_loss: 0.015397 | psnr: 18.125666 | perceptual_loss: 0.414479 | norm_perceptual_loss: 26.919653 [13:25:20.870371] [Epoch 1] | Iter: 31250 | Optim: 31250 | Time: 0.75s | LR: 0.000228 | SkipOpt: 0 loss: 0.200104 | l2_loss: 0.010330 | psnr: 19.858839 | perceptual_loss: 0.379547 | norm_perceptual_loss: 36.740902 [13:26:17.560445] [Epoch 1] | Iter: 31300 | Optim: 31300 | Time: 1.07s | LR: 0.000227 | SkipOpt: 0 loss: 0.193769 | l2_loss: 0.011220 | psnr: 19.500246 | perceptual_loss: 0.365098 | norm_perceptual_loss: 32.541271 [13:27:20.022214] [Epoch 1] | Iter: 31350 | Optim: 31350 | Time: 1.07s | LR: 0.000227 | SkipOpt: 0 loss: 0.202323 | l2_loss: 0.011272 | psnr: 19.480059 | perceptual_loss: 0.382103 | norm_perceptual_loss: 33.898937 [13:28:26.061739] [Epoch 1] | Iter: 31400 | Optim: 31400 | Time: 1.08s | LR: 0.000226 | SkipOpt: 0 loss: 0.206629 | l2_loss: 0.013442 | psnr: 18.715452 | perceptual_loss: 0.386375 | norm_perceptual_loss: 28.744429 [13:29:29.224034] [Epoch 1] | Iter: 31450 | Optim: 31450 | Time: 1.08s | LR: 0.000226 | SkipOpt: 0 loss: 0.179456 | l2_loss: 0.011445 | psnr: 19.413788 | perceptual_loss: 0.336021 | norm_perceptual_loss: 29.359270 [13:30:33.590899] [Epoch 1] | Iter: 31500 | Optim: 31500 | Time: 0.75s | LR: 0.000225 | SkipOpt: 0 loss: 0.159136 | l2_loss: 0.006481 | psnr: 21.883484 | perceptual_loss: 0.305310 | norm_perceptual_loss: 47.107452 [13:31:35.547245] [Epoch 1] | Iter: 31550 | Optim: 31550 | Time: 1.27s | LR: 0.000225 | SkipOpt: 0 loss: 0.176090 | l2_loss: 0.011103 | psnr: 19.545458 | perceptual_loss: 0.329973 | norm_perceptual_loss: 29.718334 [13:32:37.017526] [Epoch 1] | Iter: 31600 | Optim: 31600 | Time: 0.75s | LR: 0.000224 | SkipOpt: 0 loss: 0.258594 | l2_loss: 0.016108 | psnr: 17.929531 | perceptual_loss: 0.484971 | norm_perceptual_loss: 30.107090 [13:33:21.445630] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [13:33:40.135231] [Epoch 1] | Iter: 31650 | Optim: 31650 | Time: 0.75s | LR: 0.000224 | SkipOpt: 0 loss: 0.146675 | l2_loss: 0.006408 | psnr: 21.932930 | perceptual_loss: 0.280535 | norm_perceptual_loss: 43.780460 [13:34:41.633367] [Epoch 1] | Iter: 31700 | Optim: 31700 | Time: 1.91s | LR: 0.000223 | SkipOpt: 0 loss: 0.215300 | l2_loss: 0.014610 | psnr: 18.353493 | perceptual_loss: 0.401379 | norm_perceptual_loss: 27.472872 [13:35:47.254228] [Epoch 1] | Iter: 31750 | Optim: 31750 | Time: 0.75s | LR: 0.000223 | SkipOpt: 0 loss: 0.191474 | l2_loss: 0.012347 | psnr: 19.084459 | perceptual_loss: 0.358255 | norm_perceptual_loss: 29.016012 [13:36:45.547928] [Epoch 1] | Iter: 31800 | Optim: 31800 | Time: 1.08s | LR: 0.000222 | SkipOpt: 0 loss: 0.203574 | l2_loss: 0.013362 | psnr: 18.741426 | perceptual_loss: 0.380424 | norm_perceptual_loss: 28.471514 [13:37:47.512248] [Epoch 1] | Iter: 31850 | Optim: 31850 | Time: 1.91s | LR: 0.000222 | SkipOpt: 0 loss: 0.204988 | l2_loss: 0.014881 | psnr: 18.273632 | perceptual_loss: 0.380214 | norm_perceptual_loss: 25.550049 [13:38:47.359236] [Epoch 1] | Iter: 31900 | Optim: 31900 | Time: 1.07s | LR: 0.000221 | SkipOpt: 0 loss: 0.215567 | l2_loss: 0.014455 | psnr: 18.399837 | perceptual_loss: 0.402224 | norm_perceptual_loss: 27.826054 [13:39:48.947683] [Epoch 1] | Iter: 31950 | Optim: 31950 | Time: 0.75s | LR: 0.000221 | SkipOpt: 0 loss: 0.168976 | l2_loss: 0.006970 | psnr: 21.567755 | perceptual_loss: 0.324012 | norm_perceptual_loss: 46.487560 [13:40:54.409508] [Epoch 1] | Iter: 32000 | Optim: 32000 | Time: 1.08s | LR: 0.000220 | SkipOpt: 0 loss: 0.194622 | l2_loss: 0.010762 | psnr: 19.681013 | perceptual_loss: 0.367720 | norm_perceptual_loss: 34.167877 [13:41:01.247166] Saved checkpoint at iteration 32000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [13:41:01.247775] Saved latest checkpoint at iteration 32000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [13:42:03.391837] [Epoch 1] | Iter: 32050 | Optim: 32050 | Time: 1.08s | LR: 0.000220 | SkipOpt: 0 loss: 0.213224 | l2_loss: 0.014628 | psnr: 18.348106 | perceptual_loss: 0.397193 | norm_perceptual_loss: 27.152636 [13:43:09.288633] [Epoch 1] | Iter: 32100 | Optim: 32100 | Time: 0.75s | LR: 0.000219 | SkipOpt: 0 loss: 0.194403 | l2_loss: 0.012027 | psnr: 19.198589 | perceptual_loss: 0.364752 | norm_perceptual_loss: 30.328928 [13:44:09.961573] [Epoch 1] | Iter: 32150 | Optim: 32150 | Time: 1.91s | LR: 0.000219 | SkipOpt: 0 loss: 0.225289 | l2_loss: 0.020848 | psnr: 16.809441 | perceptual_loss: 0.408883 | norm_perceptual_loss: 19.612953 [13:45:15.007647] [Epoch 1] | Iter: 32200 | Optim: 32200 | Time: 1.07s | LR: 0.000218 | SkipOpt: 0 loss: 0.224566 | l2_loss: 0.013236 | psnr: 18.782574 | perceptual_loss: 0.422661 | norm_perceptual_loss: 31.933687 [13:46:12.704964] [Epoch 1] | Iter: 32250 | Optim: 32250 | Time: 0.85s | LR: 0.000218 | SkipOpt: 0 loss: 0.189687 | l2_loss: 0.010167 | psnr: 19.927919 | perceptual_loss: 0.359039 | norm_perceptual_loss: 35.312962 [13:47:10.037762] [Epoch 1] | Iter: 32300 | Optim: 32300 | Time: 0.85s | LR: 0.000217 | SkipOpt: 0 loss: 0.211821 | l2_loss: 0.012180 | psnr: 19.143593 | perceptual_loss: 0.399282 | norm_perceptual_loss: 32.782288 [13:48:10.259555] [Epoch 1] | Iter: 32350 | Optim: 32350 | Time: 1.07s | LR: 0.000217 | SkipOpt: 0 loss: 0.218492 | l2_loss: 0.017989 | psnr: 17.449884 | perceptual_loss: 0.401006 | norm_perceptual_loss: 22.291519 [13:49:07.839413] [Epoch 1] | Iter: 32400 | Optim: 32400 | Time: 0.75s | LR: 0.000216 | SkipOpt: 0 loss: 0.174477 | l2_loss: 0.011337 | psnr: 19.454954 | perceptual_loss: 0.326281 | norm_perceptual_loss: 28.779730 [13:50:10.196878] [Epoch 1] | Iter: 32450 | Optim: 32450 | Time: 1.27s | LR: 0.000216 | SkipOpt: 0 loss: 0.186606 | l2_loss: 0.012981 | psnr: 18.866840 | perceptual_loss: 0.347250 | norm_perceptual_loss: 26.750120 [13:51:11.393167] [Epoch 1] | Iter: 32500 | Optim: 32500 | Time: 1.91s | LR: 0.000215 | SkipOpt: 0 loss: 0.206432 | l2_loss: 0.014931 | psnr: 18.258974 | perceptual_loss: 0.383000 | norm_perceptual_loss: 25.650536 [13:52:16.042698] [Epoch 1] | Iter: 32550 | Optim: 32550 | Time: 1.91s | LR: 0.000215 | SkipOpt: 0 loss: 0.193380 | l2_loss: 0.011708 | psnr: 19.315241 | perceptual_loss: 0.363345 | norm_perceptual_loss: 31.034388 [13:53:16.012365] [Epoch 1] | Iter: 32600 | Optim: 32600 | Time: 1.91s | LR: 0.000214 | SkipOpt: 0 loss: 0.200887 | l2_loss: 0.014180 | psnr: 18.483389 | perceptual_loss: 0.373415 | norm_perceptual_loss: 26.334871 [13:54:18.130415] [Epoch 1] | Iter: 32650 | Optim: 32650 | Time: 1.07s | LR: 0.000214 | SkipOpt: 0 loss: 0.185138 | l2_loss: 0.010412 | psnr: 19.824497 | perceptual_loss: 0.349452 | norm_perceptual_loss: 33.561188 [13:55:19.057298] [Epoch 1] | Iter: 32700 | Optim: 32700 | Time: 1.92s | LR: 0.000213 | SkipOpt: 0 loss: 0.234135 | l2_loss: 0.015732 | psnr: 18.032040 | perceptual_loss: 0.436804 | norm_perceptual_loss: 27.764576 [13:56:20.260777] [Epoch 1] | Iter: 32750 | Optim: 32750 | Time: 1.07s | LR: 0.000213 | SkipOpt: 0 loss: 0.204035 | l2_loss: 0.014008 | psnr: 18.536343 | perceptual_loss: 0.380054 | norm_perceptual_loss: 27.131834 [13:57:23.577620] [Epoch 1] | Iter: 32800 | Optim: 32800 | Time: 0.76s | LR: 0.000212 | SkipOpt: 0 loss: 0.161624 | l2_loss: 0.010131 | psnr: 19.943401 | perceptual_loss: 0.302986 | norm_perceptual_loss: 29.906281 [13:58:25.843465] [Epoch 1] | Iter: 32850 | Optim: 32850 | Time: 1.07s | LR: 0.000212 | SkipOpt: 0 loss: 0.171095 | l2_loss: 0.009288 | psnr: 20.320852 | perceptual_loss: 0.323615 | norm_perceptual_loss: 34.842857 [13:58:59.394183] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [13:59:30.270175] [Epoch 1] | Iter: 32900 | Optim: 32900 | Time: 1.07s | LR: 0.000211 | SkipOpt: 0 loss: 0.175840 | l2_loss: 0.007151 | psnr: 21.456047 | perceptual_loss: 0.337378 | norm_perceptual_loss: 47.176003 [14:00:32.909640] [Epoch 1] | Iter: 32950 | Optim: 32950 | Time: 0.76s | LR: 0.000211 | SkipOpt: 0 loss: 0.175250 | l2_loss: 0.009811 | psnr: 20.082874 | perceptual_loss: 0.330878 | norm_perceptual_loss: 33.725269 [14:01:34.638913] [Epoch 1] | Iter: 33000 | Optim: 33000 | Time: 0.75s | LR: 0.000210 | SkipOpt: 0 loss: 0.149526 | l2_loss: 0.009199 | psnr: 20.362482 | perceptual_loss: 0.280653 | norm_perceptual_loss: 30.508305 [14:01:41.177919] Saved checkpoint at iteration 33000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [14:01:41.178341] Saved latest checkpoint at iteration 33000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [14:02:40.331858] [Epoch 1] | Iter: 33050 | Optim: 33050 | Time: 0.75s | LR: 0.000210 | SkipOpt: 0 loss: 0.167744 | l2_loss: 0.010581 | psnr: 19.754787 | perceptual_loss: 0.314326 | norm_perceptual_loss: 29.707001 [14:03:45.494843] [Epoch 1] | Iter: 33100 | Optim: 33100 | Time: 0.75s | LR: 0.000209 | SkipOpt: 0 loss: 0.224580 | l2_loss: 0.014804 | psnr: 18.296206 | perceptual_loss: 0.419551 | norm_perceptual_loss: 28.340372 [14:04:43.586104] [Epoch 1] | Iter: 33150 | Optim: 33150 | Time: 1.07s | LR: 0.000209 | SkipOpt: 0 loss: 0.186132 | l2_loss: 0.009436 | psnr: 20.251944 | perceptual_loss: 0.353391 | norm_perceptual_loss: 37.449783 [14:05:50.714137] [Epoch 1] | Iter: 33200 | Optim: 33200 | Time: 1.08s | LR: 0.000208 | SkipOpt: 0 loss: 0.179689 | l2_loss: 0.011072 | psnr: 19.557848 | perceptual_loss: 0.337234 | norm_perceptual_loss: 30.459017 [14:06:48.626114] [Epoch 1] | Iter: 33250 | Optim: 33250 | Time: 1.08s | LR: 0.000208 | SkipOpt: 0 loss: 0.177637 | l2_loss: 0.013932 | psnr: 18.559805 | perceptual_loss: 0.327409 | norm_perceptual_loss: 23.500191 [14:07:43.858804] [Epoch 1] | Iter: 33300 | Optim: 33300 | Time: 1.90s | LR: 0.000207 | SkipOpt: 0 loss: 0.203835 | l2_loss: 0.013553 | psnr: 18.679565 | perceptual_loss: 0.380564 | norm_perceptual_loss: 28.079157 [14:08:48.020568] [Epoch 1] | Iter: 33350 | Optim: 33350 | Time: 0.75s | LR: 0.000207 | SkipOpt: 0 loss: 0.142570 | l2_loss: 0.008206 | psnr: 20.858652 | perceptual_loss: 0.268727 | norm_perceptual_loss: 32.747440 [14:09:44.254902] [Epoch 1] | Iter: 33400 | Optim: 33400 | Time: 1.07s | LR: 0.000206 | SkipOpt: 0 loss: 0.177110 | l2_loss: 0.010584 | psnr: 19.753300 | perceptual_loss: 0.333050 | norm_perceptual_loss: 31.465878 [14:10:52.831503] [Epoch 1] | Iter: 33450 | Optim: 33450 | Time: 1.08s | LR: 0.000206 | SkipOpt: 0 loss: 0.190921 | l2_loss: 0.010560 | psnr: 19.763294 | perceptual_loss: 0.360721 | norm_perceptual_loss: 34.158676 [14:11:54.567312] [Epoch 1] | Iter: 33500 | Optim: 33500 | Time: 0.85s | LR: 0.000205 | SkipOpt: 0 loss: 0.209822 | l2_loss: 0.016261 | psnr: 17.888531 | perceptual_loss: 0.387121 | norm_perceptual_loss: 23.806738 [14:12:59.760708] [Epoch 1] | Iter: 33550 | Optim: 33550 | Time: 0.75s | LR: 0.000205 | SkipOpt: 0 loss: 0.173350 | l2_loss: 0.008040 | psnr: 20.947657 | perceptual_loss: 0.330622 | norm_perceptual_loss: 41.124157 [14:14:03.969611] [Epoch 1] | Iter: 33600 | Optim: 33600 | Time: 1.08s | LR: 0.000204 | SkipOpt: 0 loss: 0.174017 | l2_loss: 0.008554 | psnr: 20.678326 | perceptual_loss: 0.330926 | norm_perceptual_loss: 38.686806 [14:15:01.991386] [Epoch 1] | Iter: 33650 | Optim: 33650 | Time: 1.91s | LR: 0.000204 | SkipOpt: 0 loss: 0.225286 | l2_loss: 0.016686 | psnr: 17.776554 | perceptual_loss: 0.417201 | norm_perceptual_loss: 25.003479 [14:16:08.843726] [Epoch 1] | Iter: 33700 | Optim: 33700 | Time: 0.75s | LR: 0.000203 | SkipOpt: 0 loss: 0.193725 | l2_loss: 0.009830 | psnr: 20.074562 | perceptual_loss: 0.367791 | norm_perceptual_loss: 37.415985 [14:17:09.992649] [Epoch 1] | Iter: 33750 | Optim: 33750 | Time: 0.75s | LR: 0.000203 | SkipOpt: 0 loss: 0.194776 | l2_loss: 0.011572 | psnr: 19.365847 | perceptual_loss: 0.366407 | norm_perceptual_loss: 31.662735 [14:18:11.675416] [Epoch 1] | Iter: 33800 | Optim: 33800 | Time: 1.91s | LR: 0.000202 | SkipOpt: 0 loss: 0.213778 | l2_loss: 0.015266 | psnr: 18.162836 | perceptual_loss: 0.397025 | norm_perceptual_loss: 26.007668 [14:19:08.222062] [Epoch 1] | Iter: 33850 | Optim: 33850 | Time: 0.75s | LR: 0.000202 | SkipOpt: 0 loss: 0.161156 | l2_loss: 0.008471 | psnr: 20.720863 | perceptual_loss: 0.305370 | norm_perceptual_loss: 36.050613 [14:20:09.901136] [Epoch 1] | Iter: 33900 | Optim: 33900 | Time: 0.75s | LR: 0.000201 | SkipOpt: 0 loss: 0.154668 | l2_loss: 0.007032 | psnr: 21.529104 | perceptual_loss: 0.295272 | norm_perceptual_loss: 41.988743 [14:21:14.764889] [Epoch 1] | Iter: 33950 | Optim: 33950 | Time: 0.76s | LR: 0.000201 | SkipOpt: 0 loss: 0.177417 | l2_loss: 0.007747 | psnr: 21.108465 | perceptual_loss: 0.339340 | norm_perceptual_loss: 43.800755 [14:22:18.929397] [Epoch 1] | Iter: 34000 | Optim: 34000 | Time: 1.91s | LR: 0.000200 | SkipOpt: 0 loss: 0.206098 | l2_loss: 0.016683 | psnr: 17.777237 | perceptual_loss: 0.378830 | norm_perceptual_loss: 22.707443 [14:22:25.402436] Saved checkpoint at iteration 34000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [14:22:25.402815] Saved latest checkpoint at iteration 34000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [14:23:24.501809] [Epoch 1] | Iter: 34050 | Optim: 34050 | Time: 1.91s | LR: 0.000199 | SkipOpt: 0 loss: 0.235880 | l2_loss: 0.019686 | psnr: 17.058424 | perceptual_loss: 0.432388 | norm_perceptual_loss: 21.964243 [14:24:23.727780] [Epoch 1] | Iter: 34100 | Optim: 34100 | Time: 1.07s | LR: 0.000199 | SkipOpt: 0 loss: 0.185670 | l2_loss: 0.011174 | psnr: 19.517935 | perceptual_loss: 0.348992 | norm_perceptual_loss: 31.232641 [14:25:24.743866] [Epoch 1] | Iter: 34150 | Optim: 34150 | Time: 1.07s | LR: 0.000198 | SkipOpt: 0 loss: 0.244919 | l2_loss: 0.020502 | psnr: 16.882048 | perceptual_loss: 0.448834 | norm_perceptual_loss: 21.892246 [14:26:25.869556] [Epoch 1] | Iter: 34200 | Optim: 34200 | Time: 0.75s | LR: 0.000198 | SkipOpt: 0 loss: 0.195623 | l2_loss: 0.012485 | psnr: 19.036049 | perceptual_loss: 0.366276 | norm_perceptual_loss: 29.336803 [14:27:25.844715] [Epoch 1] | Iter: 34250 | Optim: 34250 | Time: 1.91s | LR: 0.000197 | SkipOpt: 0 loss: 0.207452 | l2_loss: 0.016632 | psnr: 17.790598 | perceptual_loss: 0.381641 | norm_perceptual_loss: 22.946405 [14:28:28.608703] [Epoch 1] | Iter: 34300 | Optim: 34300 | Time: 1.07s | LR: 0.000197 | SkipOpt: 0 loss: 0.276291 | l2_loss: 0.023071 | psnr: 16.369318 | perceptual_loss: 0.506440 | norm_perceptual_loss: 21.951286 [14:29:31.627092] [Epoch 1] | Iter: 34350 | Optim: 34350 | Time: 1.07s | LR: 0.000196 | SkipOpt: 0 loss: 0.219335 | l2_loss: 0.017699 | psnr: 17.520515 | perceptual_loss: 0.403271 | norm_perceptual_loss: 22.784996 [14:30:27.969788] [Epoch 1] | Iter: 34400 | Optim: 34400 | Time: 1.07s | LR: 0.000196 | SkipOpt: 0 loss: 0.223665 | l2_loss: 0.019149 | psnr: 17.178577 | perceptual_loss: 0.409032 | norm_perceptual_loss: 21.360693 [14:31:27.174671] [Epoch 1] | Iter: 34450 | Optim: 34450 | Time: 1.91s | LR: 0.000195 | SkipOpt: 0 loss: 0.182621 | l2_loss: 0.013230 | psnr: 18.784405 | perceptual_loss: 0.338782 | norm_perceptual_loss: 25.607122 [14:32:30.610082] [Epoch 1] | Iter: 34500 | Optim: 34500 | Time: 1.91s | LR: 0.000195 | SkipOpt: 0 loss: 0.223426 | l2_loss: 0.017733 | psnr: 17.512281 | perceptual_loss: 0.411387 | norm_perceptual_loss: 23.199497 [14:33:35.402940] [Epoch 1] | Iter: 34550 | Optim: 34550 | Time: 1.91s | LR: 0.000194 | SkipOpt: 0 loss: 0.222411 | l2_loss: 0.017142 | psnr: 17.659348 | perceptual_loss: 0.410538 | norm_perceptual_loss: 23.949062 [14:34:35.785753] [Epoch 1] | Iter: 34600 | Optim: 34600 | Time: 1.07s | LR: 0.000194 | SkipOpt: 0 loss: 0.183075 | l2_loss: 0.009929 | psnr: 20.030792 | perceptual_loss: 0.346291 | norm_perceptual_loss: 34.875473 [14:35:38.362912] [Epoch 1] | Iter: 34650 | Optim: 34650 | Time: 1.07s | LR: 0.000193 | SkipOpt: 0 loss: 0.185689 | l2_loss: 0.009569 | psnr: 20.191154 | perceptual_loss: 0.352240 | norm_perceptual_loss: 36.808994 [14:35:56.022435] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [14:36:41.268931] [Epoch 1] | Iter: 34700 | Optim: 34700 | Time: 0.85s | LR: 0.000193 | SkipOpt: 0 loss: 0.164091 | l2_loss: 0.009876 | psnr: 20.054295 | perceptual_loss: 0.308431 | norm_perceptual_loss: 31.231159 [14:37:43.625723] [Epoch 1] | Iter: 34750 | Optim: 34750 | Time: 1.07s | LR: 0.000192 | SkipOpt: 0 loss: 0.197730 | l2_loss: 0.015060 | psnr: 18.221853 | perceptual_loss: 0.365341 | norm_perceptual_loss: 24.259638 [14:38:41.880933] [Epoch 1] | Iter: 34800 | Optim: 34800 | Time: 0.75s | LR: 0.000192 | SkipOpt: 0 loss: 0.173798 | l2_loss: 0.008082 | psnr: 20.924812 | perceptual_loss: 0.331431 | norm_perceptual_loss: 41.008591 [14:39:38.493047] [Epoch 1] | Iter: 34850 | Optim: 34850 | Time: 1.07s | LR: 0.000191 | SkipOpt: 0 loss: 0.228498 | l2_loss: 0.015010 | psnr: 18.236212 | perceptual_loss: 0.426976 | norm_perceptual_loss: 28.446188 [14:40:40.910571] [Epoch 1] | Iter: 34900 | Optim: 34900 | Time: 1.91s | LR: 0.000191 | SkipOpt: 0 loss: 0.214560 | l2_loss: 0.014733 | psnr: 18.317055 | perceptual_loss: 0.399653 | norm_perceptual_loss: 27.126184 [14:41:43.832701] [Epoch 1] | Iter: 34950 | Optim: 34950 | Time: 1.07s | LR: 0.000190 | SkipOpt: 0 loss: 0.178672 | l2_loss: 0.013811 | psnr: 18.597614 | perceptual_loss: 0.329721 | norm_perceptual_loss: 23.873030 [14:42:49.563644] [Epoch 1] | Iter: 35000 | Optim: 35000 | Time: 0.75s | LR: 0.000190 | SkipOpt: 0 loss: 0.160169 | l2_loss: 0.006614 | psnr: 21.795170 | perceptual_loss: 0.307109 | norm_perceptual_loss: 46.431122 [14:42:51.826544] Found 140 sequences [14:43:16.188855] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [14:43:17.089068] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [14:44:08.546037] Gathering scores from all processes... [14:44:08.546985] Padding local tensor of length 140 to max count: 140 [14:44:08.547566] Padding local tensor of length 140 to max count: 140 [14:44:08.548039] Padding local tensor of length 140 to max count: 140 [14:44:08.548309] Gathering scene names from all processes... [14:44:08.548425] Padding local tensor of length 140 on rank 0 to max count: 140 [14:44:08.549522] Evaluation metrics: [14:44:08.549689] psnr: 16.9545 [14:44:08.549768] ssim: 0.4740 [14:44:08.549840] lpips: 0.5025 [14:44:08.553638] Found 140 sequences [14:44:33.037452] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [14:44:33.888683] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [14:45:26.570253] Gathering scores from all processes... [14:45:26.571110] Padding local tensor of length 140 to max count: 140 [14:45:26.571654] Padding local tensor of length 140 to max count: 140 [14:45:26.572142] Padding local tensor of length 140 to max count: 140 [14:45:26.572469] Gathering scene names from all processes... [14:45:26.572616] Padding local tensor of length 140 on rank 0 to max count: 140 [14:45:26.573693] Evaluation metrics: [14:45:26.573891] psnr: 19.5580 [14:45:26.573995] ssim: 0.5559 [14:45:26.574077] lpips: 0.4063 [14:45:26.577511] Found 140 sequences [14:45:52.831572] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [14:45:53.676499] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [14:46:49.205753] Gathering scores from all processes... [14:46:49.206964] Padding local tensor of length 140 to max count: 140 [14:46:49.207649] Padding local tensor of length 140 to max count: 140 [14:46:49.208155] Padding local tensor of length 140 to max count: 140 [14:46:49.208434] Gathering scene names from all processes... [14:46:49.208552] Padding local tensor of length 140 on rank 0 to max count: 140 [14:46:49.209497] Evaluation metrics: [14:46:49.218244] psnr: 20.5976 [14:46:49.218335] ssim: 0.5891 [14:46:49.218409] lpips: 0.3723 [14:46:49.219886] [Eval][iter 35000] dl3dv_2view | psnr: 16.954477 | ssim: 0.474047 | lpips: 0.502500 || dl3dv_4view | psnr: 19.558046 | ssim: 0.555949 | lpips: 0.406279 || dl3dv_6view | psnr: 20.597551 | ssim: 0.589130 | lpips: 0.372302 [14:46:49.222746] [Eval][iter 35000] dl3dv_2view | psnr: 16.954477 | ssim: 0.474047 | lpips: 0.502500 || dl3dv_4view | psnr: 19.558046 | ssim: 0.555949 | lpips: 0.406279 || dl3dv_6view | psnr: 20.597551 | ssim: 0.589130 | lpips: 0.372302 [14:46:54.016580] Saved checkpoint at iteration 35000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [14:46:54.017216] Saved latest checkpoint at iteration 35000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [14:47:49.181107] [Epoch 1] | Iter: 35050 | Optim: 35050 | Time: 0.81s | LR: 0.000189 | SkipOpt: 0 loss: 0.226469 | l2_loss: 0.016634 | psnr: 17.789967 | perceptual_loss: 0.419669 | norm_perceptual_loss: 25.229202 [14:48:49.075867] [Epoch 1] | Iter: 35100 | Optim: 35100 | Time: 1.91s | LR: 0.000189 | SkipOpt: 0 loss: 0.204257 | l2_loss: 0.015716 | psnr: 18.036545 | perceptual_loss: 0.377081 | norm_perceptual_loss: 23.993238 [14:49:48.464022] [Epoch 1] | Iter: 35150 | Optim: 35150 | Time: 0.75s | LR: 0.000188 | SkipOpt: 0 loss: 0.196221 | l2_loss: 0.010848 | psnr: 19.646385 | perceptual_loss: 0.370745 | norm_perceptual_loss: 34.175388 [14:50:50.658209] [Epoch 1] | Iter: 35200 | Optim: 35200 | Time: 1.91s | LR: 0.000188 | SkipOpt: 0 loss: 0.241822 | l2_loss: 0.018773 | psnr: 17.264591 | perceptual_loss: 0.446097 | norm_perceptual_loss: 23.762299 [14:51:51.750835] [Epoch 1] | Iter: 35250 | Optim: 35250 | Time: 1.27s | LR: 0.000187 | SkipOpt: 0 loss: 0.230500 | l2_loss: 0.016050 | psnr: 17.945269 | perceptual_loss: 0.428899 | norm_perceptual_loss: 26.722820 [14:52:56.638294] [Epoch 1] | Iter: 35300 | Optim: 35300 | Time: 1.08s | LR: 0.000187 | SkipOpt: 0 loss: 0.161635 | l2_loss: 0.008365 | psnr: 20.775269 | perceptual_loss: 0.306539 | norm_perceptual_loss: 36.644878 [14:53:59.078295] [Epoch 1] | Iter: 35350 | Optim: 35350 | Time: 0.75s | LR: 0.000186 | SkipOpt: 0 loss: 0.174554 | l2_loss: 0.010043 | psnr: 19.981558 | perceptual_loss: 0.329024 | norm_perceptual_loss: 32.762947 [14:54:58.940136] [Epoch 1] | Iter: 35400 | Optim: 35400 | Time: 0.75s | LR: 0.000186 | SkipOpt: 0 loss: 0.194327 | l2_loss: 0.008881 | psnr: 20.515345 | perceptual_loss: 0.370892 | norm_perceptual_loss: 41.762070 [14:56:05.438745] [Epoch 1] | Iter: 35450 | Optim: 35450 | Time: 1.90s | LR: 0.000185 | SkipOpt: 0 loss: 0.242291 | l2_loss: 0.019424 | psnr: 17.116602 | perceptual_loss: 0.445734 | norm_perceptual_loss: 22.947525 [14:57:03.513614] [Epoch 1] | Iter: 35500 | Optim: 35500 | Time: 0.75s | LR: 0.000185 | SkipOpt: 0 loss: 0.185286 | l2_loss: 0.009267 | psnr: 20.330566 | perceptual_loss: 0.352039 | norm_perceptual_loss: 37.988060 [14:58:07.521350] [Epoch 1] | Iter: 35550 | Optim: 35550 | Time: 1.07s | LR: 0.000184 | SkipOpt: 0 loss: 0.210720 | l2_loss: 0.013378 | psnr: 18.736094 | perceptual_loss: 0.394683 | norm_perceptual_loss: 29.502460 [14:59:10.265181] [Epoch 1] | Iter: 35600 | Optim: 35600 | Time: 1.07s | LR: 0.000184 | SkipOpt: 0 loss: 0.154635 | l2_loss: 0.008185 | psnr: 20.869556 | perceptual_loss: 0.292899 | norm_perceptual_loss: 35.782688 [15:00:15.332039] [Epoch 1] | Iter: 35650 | Optim: 35650 | Time: 0.75s | LR: 0.000183 | SkipOpt: 0 loss: 0.210260 | l2_loss: 0.012475 | psnr: 19.039520 | perceptual_loss: 0.395569 | norm_perceptual_loss: 31.708418 [15:01:21.452332] [Epoch 1] | Iter: 35700 | Optim: 35700 | Time: 0.75s | LR: 0.000183 | SkipOpt: 0 loss: 0.184803 | l2_loss: 0.012797 | psnr: 18.928877 | perceptual_loss: 0.344011 | norm_perceptual_loss: 26.881893 [15:02:25.839067] [Epoch 1] | Iter: 35750 | Optim: 35750 | Time: 1.91s | LR: 0.000182 | SkipOpt: 0 loss: 0.206996 | l2_loss: 0.015780 | psnr: 18.018909 | perceptual_loss: 0.382432 | norm_perceptual_loss: 24.235106 [15:03:29.386925] [Epoch 1] | Iter: 35800 | Optim: 35800 | Time: 1.91s | LR: 0.000182 | SkipOpt: 0 loss: 0.209274 | l2_loss: 0.016429 | psnr: 17.843781 | perceptual_loss: 0.385690 | norm_perceptual_loss: 23.475561 [15:04:41.725042] [Epoch 1] | Iter: 35850 | Optim: 35850 | Time: 1.91s | LR: 0.000181 | SkipOpt: 0 loss: 0.213292 | l2_loss: 0.014881 | psnr: 18.273785 | perceptual_loss: 0.396823 | norm_perceptual_loss: 26.667055 [15:05:47.332778] [Epoch 1] | Iter: 35900 | Optim: 35900 | Time: 0.85s | LR: 0.000181 | SkipOpt: 0 loss: 0.152862 | l2_loss: 0.009185 | psnr: 20.368979 | perceptual_loss: 0.287353 | norm_perceptual_loss: 31.283381 [15:06:50.471635] [Epoch 1] | Iter: 35950 | Optim: 35950 | Time: 1.91s | LR: 0.000180 | SkipOpt: 0 loss: 0.202454 | l2_loss: 0.017181 | psnr: 17.649611 | perceptual_loss: 0.370547 | norm_perceptual_loss: 21.567736 [15:07:52.320379] [Epoch 1] | Iter: 36000 | Optim: 36000 | Time: 1.91s | LR: 0.000180 | SkipOpt: 0 loss: 0.201986 | l2_loss: 0.013956 | psnr: 18.552334 | perceptual_loss: 0.376059 | norm_perceptual_loss: 26.945694 [15:07:57.608334] Saved checkpoint at iteration 36000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [15:07:57.608631] Saved latest checkpoint at iteration 36000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [15:08:54.518093] [Epoch 1] | Iter: 36050 | Optim: 36050 | Time: 0.75s | LR: 0.000179 | SkipOpt: 0 loss: 0.183051 | l2_loss: 0.010638 | psnr: 19.731224 | perceptual_loss: 0.344824 | norm_perceptual_loss: 32.413067 [15:09:50.381462] [Epoch 1] | Iter: 36100 | Optim: 36100 | Time: 1.91s | LR: 0.000179 | SkipOpt: 0 loss: 0.210818 | l2_loss: 0.017189 | psnr: 17.647583 | perceptual_loss: 0.387259 | norm_perceptual_loss: 22.529903 [15:10:55.922061] [Epoch 1] | Iter: 36150 | Optim: 36150 | Time: 0.75s | LR: 0.000178 | SkipOpt: 0 loss: 0.148197 | l2_loss: 0.008051 | psnr: 20.941360 | perceptual_loss: 0.280292 | norm_perceptual_loss: 34.813435 [15:12:00.433949] [Epoch 1] | Iter: 36200 | Optim: 36200 | Time: 0.75s | LR: 0.000178 | SkipOpt: 0 loss: 0.159749 | l2_loss: 0.007594 | psnr: 21.195301 | perceptual_loss: 0.304310 | norm_perceptual_loss: 40.072510 [15:13:02.955223] [Epoch 1] | Iter: 36250 | Optim: 36250 | Time: 1.91s | LR: 0.000177 | SkipOpt: 0 loss: 0.180724 | l2_loss: 0.012177 | psnr: 19.144432 | perceptual_loss: 0.337094 | norm_perceptual_loss: 27.681795 [15:14:01.822758] [Epoch 1] | Iter: 36300 | Optim: 36300 | Time: 1.91s | LR: 0.000177 | SkipOpt: 0 loss: 0.202304 | l2_loss: 0.015023 | psnr: 18.232441 | perceptual_loss: 0.374563 | norm_perceptual_loss: 24.932652 [15:15:03.339998] [Epoch 1] | Iter: 36350 | Optim: 36350 | Time: 0.85s | LR: 0.000176 | SkipOpt: 0 loss: 0.199646 | l2_loss: 0.010699 | psnr: 19.706392 | perceptual_loss: 0.377894 | norm_perceptual_loss: 35.319016 [15:16:05.652701] [Epoch 1] | Iter: 36400 | Optim: 36400 | Time: 1.07s | LR: 0.000176 | SkipOpt: 0 loss: 0.195123 | l2_loss: 0.012395 | psnr: 19.067398 | perceptual_loss: 0.365455 | norm_perceptual_loss: 29.483133 [15:16:59.981876] [Epoch 1] | Iter: 36450 | Optim: 36450 | Time: 0.75s | LR: 0.000175 | SkipOpt: 0 loss: 0.148678 | l2_loss: 0.006889 | psnr: 21.618294 | perceptual_loss: 0.283578 | norm_perceptual_loss: 41.162495 [15:18:06.052798] [Epoch 1] | Iter: 36500 | Optim: 36500 | Time: 1.27s | LR: 0.000175 | SkipOpt: 0 loss: 0.178062 | l2_loss: 0.010367 | psnr: 19.843262 | perceptual_loss: 0.335389 | norm_perceptual_loss: 32.350086 [15:19:09.679681] [Epoch 1] | Iter: 36550 | Optim: 36550 | Time: 0.85s | LR: 0.000174 | SkipOpt: 0 loss: 0.210533 | l2_loss: 0.013505 | psnr: 18.695028 | perceptual_loss: 0.394056 | norm_perceptual_loss: 29.178314 [15:20:15.472654] [Epoch 1] | Iter: 36600 | Optim: 36600 | Time: 1.07s | LR: 0.000174 | SkipOpt: 0 loss: 0.155912 | l2_loss: 0.008526 | psnr: 20.692682 | perceptual_loss: 0.294773 | norm_perceptual_loss: 34.574493 [15:21:18.372455] [Epoch 1] | Iter: 36650 | Optim: 36650 | Time: 1.90s | LR: 0.000173 | SkipOpt: 0 loss: 0.209642 | l2_loss: 0.015235 | psnr: 18.171474 | perceptual_loss: 0.388814 | norm_perceptual_loss: 25.520514 [15:22:21.395559] [Epoch 1] | Iter: 36700 | Optim: 36700 | Time: 1.07s | LR: 0.000173 | SkipOpt: 0 loss: 0.206651 | l2_loss: 0.012945 | psnr: 18.878880 | perceptual_loss: 0.387411 | norm_perceptual_loss: 29.926754 [15:23:16.171891] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [15:23:18.537615] [Epoch 1] | Iter: 36750 | Optim: 36750 | Time: 1.07s | LR: 0.000172 | SkipOpt: 0 loss: 0.190042 | l2_loss: 0.012959 | psnr: 18.874147 | perceptual_loss: 0.354166 | norm_perceptual_loss: 27.328852 [15:24:23.862811] [Epoch 1] | Iter: 36800 | Optim: 36800 | Time: 1.91s | LR: 0.000172 | SkipOpt: 0 loss: 0.215597 | l2_loss: 0.016998 | psnr: 17.696035 | perceptual_loss: 0.397199 | norm_perceptual_loss: 23.367451 [15:25:20.738956] [Epoch 1] | Iter: 36850 | Optim: 36850 | Time: 1.07s | LR: 0.000171 | SkipOpt: 0 loss: 0.189292 | l2_loss: 0.011032 | psnr: 19.573311 | perceptual_loss: 0.356518 | norm_perceptual_loss: 32.315643 [15:26:21.003227] [Epoch 1] | Iter: 36900 | Optim: 36900 | Time: 1.91s | LR: 0.000171 | SkipOpt: 0 loss: 0.193321 | l2_loss: 0.013600 | psnr: 18.664486 | perceptual_loss: 0.359442 | norm_perceptual_loss: 26.428787 [15:27:20.759411] [Epoch 1] | Iter: 36950 | Optim: 36950 | Time: 1.07s | LR: 0.000170 | SkipOpt: 0 loss: 0.205875 | l2_loss: 0.013703 | psnr: 18.631851 | perceptual_loss: 0.384345 | norm_perceptual_loss: 28.048273 [15:28:22.192558] [Epoch 1] | Iter: 37000 | Optim: 37000 | Time: 0.85s | LR: 0.000170 | SkipOpt: 0 loss: 0.169580 | l2_loss: 0.007317 | psnr: 21.356922 | perceptual_loss: 0.324528 | norm_perceptual_loss: 44.355106 [15:28:28.052806] Saved checkpoint at iteration 37000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [15:28:28.053222] Saved latest checkpoint at iteration 37000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [15:29:35.245219] [Epoch 1] | Iter: 37050 | Optim: 37050 | Time: 1.07s | LR: 0.000169 | SkipOpt: 0 loss: 0.242016 | l2_loss: 0.018403 | psnr: 17.351210 | perceptual_loss: 0.447226 | norm_perceptual_loss: 24.302334 [15:30:37.870101] [Epoch 1] | Iter: 37100 | Optim: 37100 | Time: 1.07s | LR: 0.000169 | SkipOpt: 0 loss: 0.199680 | l2_loss: 0.011251 | psnr: 19.487989 | perceptual_loss: 0.376857 | norm_perceptual_loss: 33.494652 [15:31:44.132319] [Epoch 1] | Iter: 37150 | Optim: 37150 | Time: 0.75s | LR: 0.000168 | SkipOpt: 0 loss: 0.160499 | l2_loss: 0.006896 | psnr: 21.613735 | perceptual_loss: 0.307205 | norm_perceptual_loss: 44.545269 [15:32:50.154868] [Epoch 1] | Iter: 37200 | Optim: 37200 | Time: 0.75s | LR: 0.000168 | SkipOpt: 0 loss: 0.168079 | l2_loss: 0.009295 | psnr: 20.317448 | perceptual_loss: 0.317568 | norm_perceptual_loss: 34.165005 [15:33:54.325574] [Epoch 1] | Iter: 37250 | Optim: 37250 | Time: 1.91s | LR: 0.000167 | SkipOpt: 0 loss: 0.196881 | l2_loss: 0.014287 | psnr: 18.450542 | perceptual_loss: 0.365187 | norm_perceptual_loss: 25.560509 [15:34:58.425136] [Epoch 1] | Iter: 37300 | Optim: 37300 | Time: 1.08s | LR: 0.000167 | SkipOpt: 0 loss: 0.191097 | l2_loss: 0.015638 | psnr: 18.058083 | perceptual_loss: 0.350917 | norm_perceptual_loss: 22.439451 [15:36:00.645251] [Epoch 1] | Iter: 37350 | Optim: 37350 | Time: 1.91s | LR: 0.000166 | SkipOpt: 0 loss: 0.218011 | l2_loss: 0.018434 | psnr: 17.343752 | perceptual_loss: 0.399153 | norm_perceptual_loss: 21.652832 [15:37:07.185666] [Epoch 1] | Iter: 37400 | Optim: 37400 | Time: 0.75s | LR: 0.000166 | SkipOpt: 0 loss: 0.213954 | l2_loss: 0.017304 | psnr: 17.618614 | perceptual_loss: 0.393301 | norm_perceptual_loss: 22.729343 [15:38:15.320982] [Epoch 1] | Iter: 37450 | Optim: 37450 | Time: 1.91s | LR: 0.000165 | SkipOpt: 0 loss: 0.199365 | l2_loss: 0.014512 | psnr: 18.382862 | perceptual_loss: 0.369707 | norm_perceptual_loss: 25.476711 [15:39:21.448303] [Epoch 1] | Iter: 37500 | Optim: 37500 | Time: 0.85s | LR: 0.000165 | SkipOpt: 0 loss: 0.213004 | l2_loss: 0.016068 | psnr: 17.940351 | perceptual_loss: 0.393872 | norm_perceptual_loss: 24.512629 [15:40:24.514534] [Epoch 1] | Iter: 37550 | Optim: 37550 | Time: 1.07s | LR: 0.000164 | SkipOpt: 0 loss: 0.206467 | l2_loss: 0.015704 | psnr: 18.039948 | perceptual_loss: 0.381525 | norm_perceptual_loss: 24.295065 [15:41:29.594050] [Epoch 1] | Iter: 37600 | Optim: 37600 | Time: 1.07s | LR: 0.000164 | SkipOpt: 0 loss: 0.186968 | l2_loss: 0.012136 | psnr: 19.159164 | perceptual_loss: 0.349663 | norm_perceptual_loss: 28.811543 [15:42:35.910862] [Epoch 1] | Iter: 37650 | Optim: 37650 | Time: 0.86s | LR: 0.000163 | SkipOpt: 0 loss: 0.201350 | l2_loss: 0.011969 | psnr: 19.219324 | perceptual_loss: 0.378762 | norm_perceptual_loss: 31.644541 [15:43:41.683798] [Epoch 1] | Iter: 37700 | Optim: 37700 | Time: 1.91s | LR: 0.000163 | SkipOpt: 0 loss: 0.207649 | l2_loss: 0.016887 | psnr: 17.724440 | perceptual_loss: 0.381523 | norm_perceptual_loss: 22.592516 [15:44:45.016108] [Epoch 1] | Iter: 37750 | Optim: 37750 | Time: 1.91s | LR: 0.000162 | SkipOpt: 0 loss: 0.212440 | l2_loss: 0.017635 | psnr: 17.536299 | perceptual_loss: 0.389610 | norm_perceptual_loss: 22.093287 [15:45:45.271236] [Epoch 1] | Iter: 37800 | Optim: 37800 | Time: 1.07s | LR: 0.000162 | SkipOpt: 0 loss: 0.202994 | l2_loss: 0.014460 | psnr: 18.398190 | perceptual_loss: 0.377068 | norm_perceptual_loss: 26.075857 [15:46:45.632670] [Epoch 1] | Iter: 37850 | Optim: 37850 | Time: 1.90s | LR: 0.000161 | SkipOpt: 0 loss: 0.177334 | l2_loss: 0.011662 | psnr: 19.332388 | perceptual_loss: 0.331345 | norm_perceptual_loss: 28.413113 [15:47:46.236761] [Epoch 1] | Iter: 37900 | Optim: 37900 | Time: 1.07s | LR: 0.000161 | SkipOpt: 0 loss: 0.193955 | l2_loss: 0.011627 | psnr: 19.345274 | perceptual_loss: 0.364655 | norm_perceptual_loss: 31.362463 [15:48:49.581140] [Epoch 1] | Iter: 37950 | Optim: 37950 | Time: 0.75s | LR: 0.000160 | SkipOpt: 0 loss: 0.213428 | l2_loss: 0.011569 | psnr: 19.367031 | perceptual_loss: 0.403717 | norm_perceptual_loss: 34.896389 [15:49:49.704769] [Epoch 1] | Iter: 38000 | Optim: 38000 | Time: 0.75s | LR: 0.000160 | SkipOpt: 0 loss: 0.161841 | l2_loss: 0.010022 | psnr: 19.990242 | perceptual_loss: 0.303636 | norm_perceptual_loss: 30.295490 [15:49:56.134080] Saved checkpoint at iteration 38000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [15:49:56.134568] Saved latest checkpoint at iteration 38000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [15:50:51.943204] [Epoch 1] | Iter: 38050 | Optim: 38050 | Time: 0.75s | LR: 0.000159 | SkipOpt: 0 loss: 0.179186 | l2_loss: 0.010079 | psnr: 19.965841 | perceptual_loss: 0.338213 | norm_perceptual_loss: 33.556335 [15:51:56.649572] [Epoch 1] | Iter: 38100 | Optim: 38100 | Time: 1.91s | LR: 0.000159 | SkipOpt: 0 loss: 0.217938 | l2_loss: 0.016635 | psnr: 17.789862 | perceptual_loss: 0.402606 | norm_perceptual_loss: 24.202835 [15:52:20.491277] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [15:53:03.921232] [Epoch 1] | Iter: 38150 | Optim: 38150 | Time: 1.91s | LR: 0.000158 | SkipOpt: 0 loss: 0.182869 | l2_loss: 0.014136 | psnr: 18.496717 | perceptual_loss: 0.337466 | norm_perceptual_loss: 23.872683 [15:54:05.026049] [Epoch 1] | Iter: 38200 | Optim: 38200 | Time: 0.85s | LR: 0.000158 | SkipOpt: 0 loss: 0.184240 | l2_loss: 0.010361 | psnr: 19.846174 | perceptual_loss: 0.347759 | norm_perceptual_loss: 33.565689 [15:55:00.777898] [Epoch 1] | Iter: 38250 | Optim: 38250 | Time: 1.08s | LR: 0.000157 | SkipOpt: 0 loss: 0.213009 | l2_loss: 0.013931 | psnr: 18.560129 | perceptual_loss: 0.398156 | norm_perceptual_loss: 28.580257 [15:56:03.222954] [Epoch 1] | Iter: 38300 | Optim: 38300 | Time: 1.91s | LR: 0.000157 | SkipOpt: 0 loss: 0.233777 | l2_loss: 0.016670 | psnr: 17.780674 | perceptual_loss: 0.434215 | norm_perceptual_loss: 26.047882 [15:56:41.971726] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [15:57:05.479624] [Epoch 1] | Iter: 38350 | Optim: 38350 | Time: 1.91s | LR: 0.000156 | SkipOpt: 0 loss: 0.221112 | l2_loss: 0.016797 | psnr: 17.747602 | perceptual_loss: 0.408629 | norm_perceptual_loss: 24.327065 [15:57:56.162687] [Epoch 1] | Iter: 38400 | Optim: 38400 | Time: 1.07s | LR: 0.000156 | SkipOpt: 0 loss: 0.192024 | l2_loss: 0.009573 | psnr: 20.189457 | perceptual_loss: 0.364903 | norm_perceptual_loss: 38.117336 [15:59:00.806085] [Epoch 1] | Iter: 38450 | Optim: 38450 | Time: 0.75s | LR: 0.000155 | SkipOpt: 0 loss: 0.188992 | l2_loss: 0.012073 | psnr: 19.181759 | perceptual_loss: 0.353838 | norm_perceptual_loss: 29.307577 [16:00:03.482564] [Epoch 1] | Iter: 38500 | Optim: 38500 | Time: 0.75s | LR: 0.000155 | SkipOpt: 0 loss: 0.146096 | l2_loss: 0.005899 | psnr: 22.291983 | perceptual_loss: 0.280394 | norm_perceptual_loss: 47.529865 [16:01:02.681480] [Epoch 1] | Iter: 38550 | Optim: 38550 | Time: 1.07s | LR: 0.000154 | SkipOpt: 0 loss: 0.205607 | l2_loss: 0.014181 | psnr: 18.483013 | perceptual_loss: 0.382852 | norm_perceptual_loss: 26.998047 [16:02:01.196353] [Epoch 1] | Iter: 38600 | Optim: 38600 | Time: 1.91s | LR: 0.000154 | SkipOpt: 0 loss: 0.205514 | l2_loss: 0.014673 | psnr: 18.334826 | perceptual_loss: 0.381682 | norm_perceptual_loss: 26.012650 [16:03:01.534051] [Epoch 1] | Iter: 38650 | Optim: 38650 | Time: 0.75s | LR: 0.000153 | SkipOpt: 0 loss: 0.178852 | l2_loss: 0.009926 | psnr: 20.032135 | perceptual_loss: 0.337851 | norm_perceptual_loss: 34.036053 [16:04:02.204050] [Epoch 1] | Iter: 38700 | Optim: 38700 | Time: 1.91s | LR: 0.000153 | SkipOpt: 0 loss: 0.218613 | l2_loss: 0.015865 | psnr: 17.995594 | perceptual_loss: 0.405496 | norm_perceptual_loss: 25.559113 [16:05:00.322530] [Epoch 1] | Iter: 38750 | Optim: 38750 | Time: 1.07s | LR: 0.000152 | SkipOpt: 0 loss: 0.185673 | l2_loss: 0.010923 | psnr: 19.616581 | perceptual_loss: 0.349500 | norm_perceptual_loss: 31.996645 [16:06:03.671161] [Epoch 1] | Iter: 38800 | Optim: 38800 | Time: 0.75s | LR: 0.000152 | SkipOpt: 0 loss: 0.165087 | l2_loss: 0.008192 | psnr: 20.866028 | perceptual_loss: 0.313790 | norm_perceptual_loss: 38.303783 [16:07:06.474813] [Epoch 1] | Iter: 38850 | Optim: 38850 | Time: 0.75s | LR: 0.000151 | SkipOpt: 0 loss: 0.131859 | l2_loss: 0.007010 | psnr: 21.543056 | perceptual_loss: 0.249700 | norm_perceptual_loss: 35.622444 [16:08:07.056625] [Epoch 1] | Iter: 38900 | Optim: 38900 | Time: 0.75s | LR: 0.000151 | SkipOpt: 0 loss: 0.166551 | l2_loss: 0.012114 | psnr: 19.167156 | perceptual_loss: 0.308873 | norm_perceptual_loss: 25.497410 [16:09:07.466805] [Epoch 1] | Iter: 38950 | Optim: 38950 | Time: 1.08s | LR: 0.000150 | SkipOpt: 0 loss: 0.185074 | l2_loss: 0.012276 | psnr: 19.109447 | perceptual_loss: 0.345596 | norm_perceptual_loss: 28.152273 [16:10:10.059241] [Epoch 1] | Iter: 39000 | Optim: 39000 | Time: 0.75s | LR: 0.000150 | SkipOpt: 0 loss: 0.124319 | l2_loss: 0.004133 | psnr: 23.837633 | perceptual_loss: 0.240372 | norm_perceptual_loss: 58.163055 [16:10:16.405039] Saved checkpoint at iteration 39000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [16:10:16.405458] Saved latest checkpoint at iteration 39000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [16:11:14.766997] [Epoch 1] | Iter: 39050 | Optim: 39050 | Time: 1.91s | LR: 0.000149 | SkipOpt: 0 loss: 0.190524 | l2_loss: 0.012172 | psnr: 19.146305 | perceptual_loss: 0.356703 | norm_perceptual_loss: 29.304705 [16:12:21.812484] [Epoch 1] | Iter: 39100 | Optim: 39100 | Time: 1.07s | LR: 0.000149 | SkipOpt: 0 loss: 0.223040 | l2_loss: 0.015469 | psnr: 18.105244 | perceptual_loss: 0.415142 | norm_perceptual_loss: 26.836187 [16:13:28.460520] [Epoch 1] | Iter: 39150 | Optim: 39150 | Time: 1.91s | LR: 0.000148 | SkipOpt: 0 loss: 0.197479 | l2_loss: 0.012559 | psnr: 19.010406 | perceptual_loss: 0.369840 | norm_perceptual_loss: 29.447937 [16:14:25.674529] [Epoch 1] | Iter: 39200 | Optim: 39200 | Time: 1.07s | LR: 0.000148 | SkipOpt: 0 loss: 0.192099 | l2_loss: 0.011950 | psnr: 19.226236 | perceptual_loss: 0.360297 | norm_perceptual_loss: 30.149784 [16:15:24.646125] [Epoch 1] | Iter: 39250 | Optim: 39250 | Time: 1.27s | LR: 0.000147 | SkipOpt: 0 loss: 0.178306 | l2_loss: 0.011198 | psnr: 19.508724 | perceptual_loss: 0.334217 | norm_perceptual_loss: 29.847038 [16:16:18.812058] [Epoch 1] | Iter: 39300 | Optim: 39300 | Time: 1.90s | LR: 0.000147 | SkipOpt: 0 loss: 0.188265 | l2_loss: 0.012049 | psnr: 19.190506 | perceptual_loss: 0.352431 | norm_perceptual_loss: 29.249945 [16:17:23.992310] [Epoch 1] | Iter: 39350 | Optim: 39350 | Time: 1.08s | LR: 0.000146 | SkipOpt: 0 loss: 0.185222 | l2_loss: 0.010776 | psnr: 19.675459 | perceptual_loss: 0.348891 | norm_perceptual_loss: 32.376972 [16:18:23.069116] [Epoch 1] | Iter: 39400 | Optim: 39400 | Time: 0.75s | LR: 0.000146 | SkipOpt: 0 loss: 0.199390 | l2_loss: 0.010543 | psnr: 19.770275 | perceptual_loss: 0.377693 | norm_perceptual_loss: 35.823406 [16:19:26.018893] [Epoch 1] | Iter: 39450 | Optim: 39450 | Time: 1.07s | LR: 0.000145 | SkipOpt: 0 loss: 0.168438 | l2_loss: 0.011236 | psnr: 19.493866 | perceptual_loss: 0.314403 | norm_perceptual_loss: 27.981667 [16:20:29.000015] [Epoch 1] | Iter: 39500 | Optim: 39500 | Time: 1.07s | LR: 0.000145 | SkipOpt: 0 loss: 0.160351 | l2_loss: 0.007834 | psnr: 21.060398 | perceptual_loss: 0.305035 | norm_perceptual_loss: 38.939411 [16:21:35.548657] [Epoch 1] | Iter: 39550 | Optim: 39550 | Time: 1.91s | LR: 0.000144 | SkipOpt: 0 loss: 0.201423 | l2_loss: 0.013672 | psnr: 18.641525 | perceptual_loss: 0.375501 | norm_perceptual_loss: 27.463970 [16:22:36.428559] [Epoch 1] | Iter: 39600 | Optim: 39600 | Time: 0.76s | LR: 0.000144 | SkipOpt: 0 loss: 0.181714 | l2_loss: 0.007302 | psnr: 21.365702 | perceptual_loss: 0.348824 | norm_perceptual_loss: 47.772270 [16:23:36.305533] [Epoch 1] | Iter: 39650 | Optim: 39650 | Time: 1.91s | LR: 0.000144 | SkipOpt: 0 loss: 0.167666 | l2_loss: 0.010093 | psnr: 19.959810 | perceptual_loss: 0.315146 | norm_perceptual_loss: 31.224308 [16:24:39.549949] [Epoch 1] | Iter: 39700 | Optim: 39700 | Time: 0.75s | LR: 0.000143 | SkipOpt: 0 loss: 0.186983 | l2_loss: 0.010344 | psnr: 19.853109 | perceptual_loss: 0.353278 | norm_perceptual_loss: 34.152874 [16:25:41.129312] [Epoch 1] | Iter: 39750 | Optim: 39750 | Time: 1.07s | LR: 0.000143 | SkipOpt: 0 loss: 0.174094 | l2_loss: 0.009143 | psnr: 20.389174 | perceptual_loss: 0.329901 | norm_perceptual_loss: 36.082901 [16:26:48.184984] [Epoch 1] | Iter: 39800 | Optim: 39800 | Time: 0.75s | LR: 0.000142 | SkipOpt: 0 loss: 0.204498 | l2_loss: 0.012214 | psnr: 19.131271 | perceptual_loss: 0.384567 | norm_perceptual_loss: 31.484659 [16:27:47.738907] [Epoch 1] | Iter: 39850 | Optim: 39850 | Time: 1.08s | LR: 0.000142 | SkipOpt: 0 loss: 0.197120 | l2_loss: 0.010926 | psnr: 19.615292 | perceptual_loss: 0.372388 | norm_perceptual_loss: 34.081940 [16:28:51.603571] [Epoch 1] | Iter: 39900 | Optim: 39900 | Time: 1.08s | LR: 0.000141 | SkipOpt: 0 loss: 0.208986 | l2_loss: 0.010103 | psnr: 19.955708 | perceptual_loss: 0.397768 | norm_perceptual_loss: 39.373169 [16:29:52.232108] [Epoch 1] | Iter: 39950 | Optim: 39950 | Time: 0.75s | LR: 0.000141 | SkipOpt: 0 loss: 0.180289 | l2_loss: 0.009678 | psnr: 20.142174 | perceptual_loss: 0.341223 | norm_perceptual_loss: 35.257816 [16:30:48.544064] [Epoch 1] | Iter: 40000 | Optim: 40000 | Time: 1.08s | LR: 0.000140 | SkipOpt: 0 loss: 0.175887 | l2_loss: 0.007986 | psnr: 20.976545 | perceptual_loss: 0.335801 | norm_perceptual_loss: 42.047119 [16:30:51.119356] Found 140 sequences [16:31:14.907207] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [16:31:15.762610] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [16:32:08.443494] Gathering scores from all processes... [16:32:08.445426] Padding local tensor of length 140 to max count: 140 [16:32:08.446779] Padding local tensor of length 140 to max count: 140 [16:32:08.447374] Padding local tensor of length 140 to max count: 140 [16:32:08.447677] Gathering scene names from all processes... [16:32:08.447800] Padding local tensor of length 140 on rank 0 to max count: 140 [16:32:08.449276] Evaluation metrics: [16:32:08.449737] psnr: 17.2705 [16:32:08.449992] ssim: 0.4850 [16:32:08.450216] lpips: 0.4938 [16:32:08.455433] Found 140 sequences [16:32:33.871636] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [16:32:34.936496] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [16:33:28.762362] Gathering scores from all processes... [16:33:28.763918] Padding local tensor of length 140 to max count: 140 [16:33:28.764962] Padding local tensor of length 140 to max count: 140 [16:33:28.765912] Padding local tensor of length 140 to max count: 140 [16:33:28.766252] Gathering scene names from all processes... [16:33:28.766558] Padding local tensor of length 140 on rank 0 to max count: 140 [16:33:28.768415] Evaluation metrics: [16:33:28.768803] psnr: 19.9364 [16:33:28.769118] ssim: 0.5708 [16:33:28.769339] lpips: 0.3932 [16:33:28.774475] Found 140 sequences [16:33:52.702922] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [16:33:53.552166] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [16:34:50.491404] Gathering scores from all processes... [16:34:50.493457] Padding local tensor of length 140 to max count: 140 [16:34:50.494443] Padding local tensor of length 140 to max count: 140 [16:34:50.495515] Padding local tensor of length 140 to max count: 140 [16:34:50.496158] Gathering scene names from all processes... [16:34:50.496325] Padding local tensor of length 140 on rank 0 to max count: 140 [16:34:50.497520] Evaluation metrics: [16:34:50.497758] psnr: 20.9575 [16:34:50.497877] ssim: 0.6047 [16:34:50.497984] lpips: 0.3568 [16:34:50.499680] [Eval][iter 40000] dl3dv_2view | psnr: 17.270464 | ssim: 0.484966 | lpips: 0.493843 || dl3dv_4view | psnr: 19.936426 | ssim: 0.570776 | lpips: 0.393177 || dl3dv_6view | psnr: 20.957508 | ssim: 0.604687 | lpips: 0.356841 [16:34:50.503967] [Eval][iter 40000] dl3dv_2view | psnr: 17.270464 | ssim: 0.484966 | lpips: 0.493843 || dl3dv_4view | psnr: 19.936426 | ssim: 0.570776 | lpips: 0.393177 || dl3dv_6view | psnr: 20.957508 | ssim: 0.604687 | lpips: 0.356841 [16:34:57.080507] Saved checkpoint at iteration 40000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [16:34:57.081355] Saved latest checkpoint at iteration 40000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [16:36:04.687646] [Epoch 1] | Iter: 40050 | Optim: 40050 | Time: 1.08s | LR: 0.000140 | SkipOpt: 0 loss: 0.161725 | l2_loss: 0.009393 | psnr: 20.272133 | perceptual_loss: 0.304664 | norm_perceptual_loss: 32.436550 [16:37:05.175833] [Epoch 1] | Iter: 40100 | Optim: 40100 | Time: 1.91s | LR: 0.000139 | SkipOpt: 0 loss: 0.224604 | l2_loss: 0.015781 | psnr: 18.018700 | perceptual_loss: 0.417646 | norm_perceptual_loss: 26.465363 [16:38:12.315094] [Epoch 1] | Iter: 40150 | Optim: 40150 | Time: 1.91s | LR: 0.000139 | SkipOpt: 0 loss: 0.220120 | l2_loss: 0.015588 | psnr: 18.072083 | perceptual_loss: 0.409063 | norm_perceptual_loss: 26.242086 [16:38:45.450986] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [16:39:08.790568] [Epoch 1] | Iter: 40200 | Optim: 40200 | Time: 0.76s | LR: 0.000138 | SkipOpt: 0 loss: 0.142059 | l2_loss: 0.005497 | psnr: 22.598640 | perceptual_loss: 0.273123 | norm_perceptual_loss: 49.684753 [16:40:03.878658] [Epoch 1] | Iter: 40250 | Optim: 40250 | Time: 0.85s | LR: 0.000138 | SkipOpt: 0 loss: 0.173616 | l2_loss: 0.010377 | psnr: 19.839195 | perceptual_loss: 0.326478 | norm_perceptual_loss: 31.461016 [16:41:02.911648] [Epoch 1] | Iter: 40300 | Optim: 40300 | Time: 1.07s | LR: 0.000137 | SkipOpt: 0 loss: 0.194180 | l2_loss: 0.014391 | psnr: 18.419044 | perceptual_loss: 0.359577 | norm_perceptual_loss: 24.985952 [16:42:09.558226] [Epoch 1] | Iter: 40350 | Optim: 40350 | Time: 0.75s | LR: 0.000137 | SkipOpt: 0 loss: 0.151558 | l2_loss: 0.007516 | psnr: 21.240284 | perceptual_loss: 0.288085 | norm_perceptual_loss: 38.330872 [16:43:12.252640] [Epoch 1] | Iter: 40400 | Optim: 40400 | Time: 1.07s | LR: 0.000136 | SkipOpt: 0 loss: 0.177830 | l2_loss: 0.012732 | psnr: 18.951077 | perceptual_loss: 0.330195 | norm_perceptual_loss: 25.934544 [16:44:09.186649] [Epoch 1] | Iter: 40450 | Optim: 40450 | Time: 1.08s | LR: 0.000136 | SkipOpt: 0 loss: 0.188266 | l2_loss: 0.011837 | psnr: 19.267687 | perceptual_loss: 0.352858 | norm_perceptual_loss: 29.810432 [16:45:12.664228] [Epoch 1] | Iter: 40500 | Optim: 40500 | Time: 0.78s | LR: 0.000135 | SkipOpt: 0 loss: 0.173671 | l2_loss: 0.009692 | psnr: 20.135921 | perceptual_loss: 0.327959 | norm_perceptual_loss: 33.838581 [16:46:13.775197] [Epoch 1] | Iter: 40550 | Optim: 40550 | Time: 1.91s | LR: 0.000135 | SkipOpt: 0 loss: 0.197669 | l2_loss: 0.014270 | psnr: 18.455902 | perceptual_loss: 0.366799 | norm_perceptual_loss: 25.705046 [16:47:16.410342] [Epoch 1] | Iter: 40600 | Optim: 40600 | Time: 0.76s | LR: 0.000134 | SkipOpt: 0 loss: 0.167136 | l2_loss: 0.008545 | psnr: 20.683096 | perceptual_loss: 0.317184 | norm_perceptual_loss: 37.121067 [16:48:19.724715] [Epoch 1] | Iter: 40650 | Optim: 40650 | Time: 0.87s | LR: 0.000134 | SkipOpt: 0 loss: 0.243319 | l2_loss: 0.017380 | psnr: 17.599529 | perceptual_loss: 0.451878 | norm_perceptual_loss: 26.000072 [16:49:16.000542] [Epoch 1] | Iter: 40700 | Optim: 40700 | Time: 0.85s | LR: 0.000133 | SkipOpt: 0 loss: 0.190205 | l2_loss: 0.014126 | psnr: 18.499826 | perceptual_loss: 0.352159 | norm_perceptual_loss: 24.929928 [16:50:17.194001] [Epoch 1] | Iter: 40750 | Optim: 40750 | Time: 1.91s | LR: 0.000133 | SkipOpt: 0 loss: 0.218118 | l2_loss: 0.016977 | psnr: 17.701380 | perceptual_loss: 0.402282 | norm_perceptual_loss: 23.695663 [16:51:20.103951] [Epoch 1] | Iter: 40800 | Optim: 40800 | Time: 0.86s | LR: 0.000132 | SkipOpt: 0 loss: 0.180203 | l2_loss: 0.009838 | psnr: 20.070774 | perceptual_loss: 0.340730 | norm_perceptual_loss: 34.632820 [16:52:22.581872] [Epoch 1] | Iter: 40850 | Optim: 40850 | Time: 0.75s | LR: 0.000132 | SkipOpt: 0 loss: 0.194172 | l2_loss: 0.011899 | psnr: 19.244926 | perceptual_loss: 0.364547 | norm_perceptual_loss: 30.636993 [16:53:21.113930] [Epoch 1] | Iter: 40900 | Optim: 40900 | Time: 1.91s | LR: 0.000131 | SkipOpt: 0 loss: 0.217093 | l2_loss: 0.015281 | psnr: 18.158352 | perceptual_loss: 0.403623 | norm_perceptual_loss: 26.412567 [16:54:26.792182] [Epoch 1] | Iter: 40950 | Optim: 40950 | Time: 1.08s | LR: 0.000131 | SkipOpt: 0 loss: 0.181026 | l2_loss: 0.010681 | psnr: 19.713800 | perceptual_loss: 0.340690 | norm_perceptual_loss: 31.896267 [16:55:30.994013] [Epoch 1] | Iter: 41000 | Optim: 41000 | Time: 1.91s | LR: 0.000131 | SkipOpt: 0 loss: 0.224279 | l2_loss: 0.019932 | psnr: 17.004515 | perceptual_loss: 0.408694 | norm_perceptual_loss: 20.504505 [16:55:36.666726] Saved checkpoint at iteration 41000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [16:55:36.667057] Saved latest checkpoint at iteration 41000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [16:56:36.404885] [Epoch 1] | Iter: 41050 | Optim: 41050 | Time: 0.75s | LR: 0.000130 | SkipOpt: 0 loss: 0.140417 | l2_loss: 0.006922 | psnr: 21.597809 | perceptual_loss: 0.266991 | norm_perceptual_loss: 38.572506 [16:57:41.556959] [Epoch 1] | Iter: 41100 | Optim: 41100 | Time: 1.91s | LR: 0.000130 | SkipOpt: 0 loss: 0.221963 | l2_loss: 0.016909 | psnr: 17.718809 | perceptual_loss: 0.410108 | norm_perceptual_loss: 24.253780 [16:58:50.000705] [Epoch 1] | Iter: 41150 | Optim: 41150 | Time: 1.07s | LR: 0.000129 | SkipOpt: 0 loss: 0.161318 | l2_loss: 0.007438 | psnr: 21.285358 | perceptual_loss: 0.307760 | norm_perceptual_loss: 41.375980 [16:59:49.662030] [Epoch 1] | Iter: 41200 | Optim: 41200 | Time: 0.75s | LR: 0.000129 | SkipOpt: 0 loss: 0.142268 | l2_loss: 0.006231 | psnr: 22.054115 | perceptual_loss: 0.272072 | norm_perceptual_loss: 43.661243 [17:00:47.326113] [Epoch 1] | Iter: 41250 | Optim: 41250 | Time: 1.91s | LR: 0.000128 | SkipOpt: 0 loss: 0.204378 | l2_loss: 0.014484 | psnr: 18.391102 | perceptual_loss: 0.379787 | norm_perceptual_loss: 26.221077 [17:01:48.071827] [Epoch 1] | Iter: 41300 | Optim: 41300 | Time: 1.27s | LR: 0.000128 | SkipOpt: 0 loss: 0.183398 | l2_loss: 0.010885 | psnr: 19.631596 | perceptual_loss: 0.345026 | norm_perceptual_loss: 31.696512 [17:02:51.280949] [Epoch 1] | Iter: 41350 | Optim: 41350 | Time: 1.90s | LR: 0.000127 | SkipOpt: 0 loss: 0.185036 | l2_loss: 0.013053 | psnr: 18.842958 | perceptual_loss: 0.343967 | norm_perceptual_loss: 26.351919 [17:03:55.131519] [Epoch 1] | Iter: 41400 | Optim: 41400 | Time: 0.76s | LR: 0.000127 | SkipOpt: 0 loss: 0.180151 | l2_loss: 0.008837 | psnr: 20.536930 | perceptual_loss: 0.342629 | norm_perceptual_loss: 38.771893 [17:04:50.102167] [Epoch 1] | Iter: 41450 | Optim: 41450 | Time: 1.08s | LR: 0.000126 | SkipOpt: 0 loss: 0.173380 | l2_loss: 0.010910 | psnr: 19.621822 | perceptual_loss: 0.324941 | norm_perceptual_loss: 29.784233 [17:05:49.752627] [Epoch 1] | Iter: 41500 | Optim: 41500 | Time: 0.75s | LR: 0.000126 | SkipOpt: 0 loss: 0.169440 | l2_loss: 0.008436 | psnr: 20.738773 | perceptual_loss: 0.322009 | norm_perceptual_loss: 38.172035 [17:06:45.530959] [Epoch 1] | Iter: 41550 | Optim: 41550 | Time: 0.76s | LR: 0.000125 | SkipOpt: 0 loss: 0.143237 | l2_loss: 0.005216 | psnr: 22.826420 | perceptual_loss: 0.276042 | norm_perceptual_loss: 52.919754 [17:07:54.922403] [Epoch 1] | Iter: 41600 | Optim: 41600 | Time: 1.91s | LR: 0.000125 | SkipOpt: 0 loss: 0.206989 | l2_loss: 0.014935 | psnr: 18.257849 | perceptual_loss: 0.384108 | norm_perceptual_loss: 25.718044 [17:08:44.002275] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [17:08:55.593388] [Epoch 1] | Iter: 41650 | Optim: 41650 | Time: 0.75s | LR: 0.000124 | SkipOpt: 0 loss: 0.128909 | l2_loss: 0.004644 | psnr: 23.330875 | perceptual_loss: 0.248530 | norm_perceptual_loss: 53.513824 [17:09:58.329810] [Epoch 1] | Iter: 41700 | Optim: 41700 | Time: 1.92s | LR: 0.000124 | SkipOpt: 0 loss: 0.236666 | l2_loss: 0.018696 | psnr: 17.282625 | perceptual_loss: 0.435941 | norm_perceptual_loss: 23.317951 [17:10:54.445296] [Epoch 1] | Iter: 41750 | Optim: 41750 | Time: 0.75s | LR: 0.000123 | SkipOpt: 0 loss: 0.186239 | l2_loss: 0.012790 | psnr: 18.931175 | perceptual_loss: 0.346897 | norm_perceptual_loss: 27.121796 [17:11:51.923169] [Epoch 1] | Iter: 41800 | Optim: 41800 | Time: 1.91s | LR: 0.000123 | SkipOpt: 0 loss: 0.204384 | l2_loss: 0.016502 | psnr: 17.824518 | perceptual_loss: 0.375763 | norm_perceptual_loss: 22.770151 [17:12:50.196640] [Epoch 1] | Iter: 41850 | Optim: 41850 | Time: 1.90s | LR: 0.000123 | SkipOpt: 0 loss: 0.188518 | l2_loss: 0.014535 | psnr: 18.375744 | perceptual_loss: 0.347965 | norm_perceptual_loss: 23.939213 [17:13:50.747460] [Epoch 1] | Iter: 41900 | Optim: 41900 | Time: 1.08s | LR: 0.000122 | SkipOpt: 0 loss: 0.197970 | l2_loss: 0.008750 | psnr: 20.580055 | perceptual_loss: 0.378441 | norm_perceptual_loss: 43.251797 [17:14:52.886407] [Epoch 1] | Iter: 41950 | Optim: 41950 | Time: 1.27s | LR: 0.000122 | SkipOpt: 0 loss: 0.180333 | l2_loss: 0.014476 | psnr: 18.393476 | perceptual_loss: 0.331715 | norm_perceptual_loss: 22.914606 [17:15:59.377427] [Epoch 1] | Iter: 42000 | Optim: 42000 | Time: 1.91s | LR: 0.000121 | SkipOpt: 0 loss: 0.231163 | l2_loss: 0.017911 | psnr: 17.468843 | perceptual_loss: 0.426505 | norm_perceptual_loss: 23.812689 [17:16:05.802052] Saved checkpoint at iteration 42000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [17:16:05.802373] Saved latest checkpoint at iteration 42000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [17:17:10.386731] [Epoch 1] | Iter: 42050 | Optim: 42050 | Time: 0.75s | LR: 0.000121 | SkipOpt: 0 loss: 0.177817 | l2_loss: 0.007267 | psnr: 21.386536 | perceptual_loss: 0.341099 | norm_perceptual_loss: 46.939056 [17:18:14.482090] [Epoch 1] | Iter: 42100 | Optim: 42100 | Time: 1.27s | LR: 0.000120 | SkipOpt: 0 loss: 0.191507 | l2_loss: 0.009916 | psnr: 20.036421 | perceptual_loss: 0.363180 | norm_perceptual_loss: 36.623898 [17:19:20.863290] [Epoch 1] | Iter: 42150 | Optim: 42150 | Time: 1.91s | LR: 0.000120 | SkipOpt: 0 loss: 0.207869 | l2_loss: 0.014009 | psnr: 18.535835 | perceptual_loss: 0.387720 | norm_perceptual_loss: 27.675901 [17:20:23.959616] [Epoch 1] | Iter: 42200 | Optim: 42200 | Time: 0.75s | LR: 0.000119 | SkipOpt: 0 loss: 0.188317 | l2_loss: 0.012877 | psnr: 18.901810 | perceptual_loss: 0.350879 | norm_perceptual_loss: 27.248215 [17:21:10.550878] New dataloader... [17:21:10.552011] Building dynamic dataloader with seed: 0 [17:21:10.625869] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.626019] Worker seed: 24691 [17:21:10.685296] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.685476] Worker seed: 24692 [17:21:10.708961] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.709111] Worker seed: 24693 [17:21:10.739532] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.739678] Worker seed: 24694 [17:21:10.770581] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.770721] Worker seed: 24695 [17:21:10.800347] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.800469] Worker seed: 24696 [17:21:10.833675] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.833828] Worker seed: 24697 [17:21:10.864942] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.865089] Worker seed: 24698 [17:21:10.896071] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.896207] Worker seed: 24699 [17:21:10.930023] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.930183] Worker seed: 24700 [17:21:10.959908] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.960053] Worker seed: 24701 [17:21:10.990963] Rank: 0, World size: 1, Distributed rank: 0 [17:21:10.991102] Worker seed: 24702 [17:21:34.557708] [Epoch 2] | Iter: 42250 | Optim: 42250 | Time: 1.91s | LR: 0.000119 | SkipOpt: 0 loss: 0.183374 | l2_loss: 0.011253 | psnr: 19.487278 | perceptual_loss: 0.344242 | norm_perceptual_loss: 30.590837 [17:22:39.264312] [Epoch 2] | Iter: 42300 | Optim: 42300 | Time: 1.08s | LR: 0.000118 | SkipOpt: 0 loss: 0.214913 | l2_loss: 0.017003 | psnr: 17.694695 | perceptual_loss: 0.395820 | norm_perceptual_loss: 23.279139 [17:23:43.423619] [Epoch 2] | Iter: 42350 | Optim: 42350 | Time: 0.85s | LR: 0.000118 | SkipOpt: 0 loss: 0.196026 | l2_loss: 0.014341 | psnr: 18.434305 | perceptual_loss: 0.363370 | norm_perceptual_loss: 25.338415 [17:24:42.386805] [Epoch 2] | Iter: 42400 | Optim: 42400 | Time: 0.76s | LR: 0.000117 | SkipOpt: 0 loss: 0.212158 | l2_loss: 0.017871 | psnr: 17.478554 | perceptual_loss: 0.388573 | norm_perceptual_loss: 21.743450 [17:25:44.792620] [Epoch 2] | Iter: 42450 | Optim: 42450 | Time: 0.85s | LR: 0.000117 | SkipOpt: 0 loss: 0.182310 | l2_loss: 0.011402 | psnr: 19.430241 | perceptual_loss: 0.341817 | norm_perceptual_loss: 29.978989 [17:26:45.617040] [Epoch 2] | Iter: 42500 | Optim: 42500 | Time: 1.08s | LR: 0.000117 | SkipOpt: 0 loss: 0.193375 | l2_loss: 0.011798 | psnr: 19.281963 | perceptual_loss: 0.363155 | norm_perceptual_loss: 30.781368 [17:27:16.188468] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [17:27:47.518014] [Epoch 2] | Iter: 42550 | Optim: 42550 | Time: 1.07s | LR: 0.000116 | SkipOpt: 0 loss: 0.172854 | l2_loss: 0.011190 | psnr: 19.511787 | perceptual_loss: 0.323328 | norm_perceptual_loss: 28.894917 [17:28:53.008492] [Epoch 2] | Iter: 42600 | Optim: 42600 | Time: 1.27s | LR: 0.000116 | SkipOpt: 0 loss: 0.202150 | l2_loss: 0.013610 | psnr: 18.661398 | perceptual_loss: 0.377080 | norm_perceptual_loss: 27.705940 [17:29:57.029543] [Epoch 2] | Iter: 42650 | Optim: 42650 | Time: 1.91s | LR: 0.000115 | SkipOpt: 0 loss: 0.228096 | l2_loss: 0.017970 | psnr: 17.454456 | perceptual_loss: 0.420252 | norm_perceptual_loss: 23.385969 [17:31:00.073726] [Epoch 2] | Iter: 42700 | Optim: 42700 | Time: 0.85s | LR: 0.000115 | SkipOpt: 0 loss: 0.166707 | l2_loss: 0.007392 | psnr: 21.312305 | perceptual_loss: 0.318631 | norm_perceptual_loss: 43.104038 [17:32:04.890337] [Epoch 2] | Iter: 42750 | Optim: 42750 | Time: 0.75s | LR: 0.000114 | SkipOpt: 0 loss: 0.145378 | l2_loss: 0.009218 | psnr: 20.353411 | perceptual_loss: 0.272320 | norm_perceptual_loss: 29.540672 [17:33:08.566521] [Epoch 2] | Iter: 42800 | Optim: 42800 | Time: 1.91s | LR: 0.000114 | SkipOpt: 0 loss: 0.213727 | l2_loss: 0.014587 | psnr: 18.360229 | perceptual_loss: 0.398280 | norm_perceptual_loss: 27.303022 [17:34:11.687262] [Epoch 2] | Iter: 42850 | Optim: 42850 | Time: 0.85s | LR: 0.000113 | SkipOpt: 0 loss: 0.204793 | l2_loss: 0.013149 | psnr: 18.811081 | perceptual_loss: 0.383287 | norm_perceptual_loss: 29.149591 [17:34:47.203868] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [17:35:14.080108] [Epoch 2] | Iter: 42900 | Optim: 42900 | Time: 1.91s | LR: 0.000113 | SkipOpt: 0 loss: 0.197026 | l2_loss: 0.013488 | psnr: 18.700636 | perceptual_loss: 0.367078 | norm_perceptual_loss: 27.215807 [17:36:13.275019] [Epoch 2] | Iter: 42950 | Optim: 42950 | Time: 1.27s | LR: 0.000112 | SkipOpt: 0 loss: 0.221782 | l2_loss: 0.020257 | psnr: 16.934240 | perceptual_loss: 0.403050 | norm_perceptual_loss: 19.896780 [17:37:15.199946] [Epoch 2] | Iter: 43000 | Optim: 43000 | Time: 0.75s | LR: 0.000112 | SkipOpt: 0 loss: 0.206740 | l2_loss: 0.012614 | psnr: 18.991396 | perceptual_loss: 0.388252 | norm_perceptual_loss: 30.778908 [17:37:20.695017] Saved checkpoint at iteration 43000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [17:37:20.695380] Saved latest checkpoint at iteration 43000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [17:38:17.720978] [Epoch 2] | Iter: 43050 | Optim: 43050 | Time: 0.75s | LR: 0.000111 | SkipOpt: 0 loss: 0.223416 | l2_loss: 0.017758 | psnr: 17.506121 | perceptual_loss: 0.411316 | norm_perceptual_loss: 23.162609 [17:38:51.592314] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [17:39:19.318114] [Epoch 2] | Iter: 43100 | Optim: 43100 | Time: 1.91s | LR: 0.000111 | SkipOpt: 0 loss: 0.180707 | l2_loss: 0.012925 | psnr: 18.885553 | perceptual_loss: 0.335564 | norm_perceptual_loss: 25.961517 [17:40:21.411287] [Epoch 2] | Iter: 43150 | Optim: 43150 | Time: 1.91s | LR: 0.000111 | SkipOpt: 0 loss: 0.196491 | l2_loss: 0.012875 | psnr: 18.902462 | perceptual_loss: 0.367232 | norm_perceptual_loss: 28.522463 [17:41:20.272842] [Epoch 2] | Iter: 43200 | Optim: 43200 | Time: 0.75s | LR: 0.000110 | SkipOpt: 0 loss: 0.158361 | l2_loss: 0.008749 | psnr: 20.580177 | perceptual_loss: 0.299224 | norm_perceptual_loss: 34.199055 [17:42:24.359284] [Epoch 2] | Iter: 43250 | Optim: 43250 | Time: 1.91s | LR: 0.000110 | SkipOpt: 0 loss: 0.218014 | l2_loss: 0.014830 | psnr: 18.288456 | perceptual_loss: 0.406367 | norm_perceptual_loss: 27.400845 [17:43:29.449718] [Epoch 2] | Iter: 43300 | Optim: 43300 | Time: 1.08s | LR: 0.000109 | SkipOpt: 0 loss: 0.187762 | l2_loss: 0.010946 | psnr: 19.607492 | perceptual_loss: 0.353632 | norm_perceptual_loss: 32.307346 [17:44:29.762677] [Epoch 2] | Iter: 43350 | Optim: 43350 | Time: 0.85s | LR: 0.000109 | SkipOpt: 0 loss: 0.179780 | l2_loss: 0.019607 | psnr: 17.075886 | perceptual_loss: 0.320347 | norm_perceptual_loss: 16.338375 [17:45:31.928490] [Epoch 2] | Iter: 43400 | Optim: 43400 | Time: 0.75s | LR: 0.000108 | SkipOpt: 0 loss: 0.164905 | l2_loss: 0.007957 | psnr: 20.992371 | perceptual_loss: 0.313896 | norm_perceptual_loss: 39.447731 [17:46:37.597177] [Epoch 2] | Iter: 43450 | Optim: 43450 | Time: 1.07s | LR: 0.000108 | SkipOpt: 0 loss: 0.188004 | l2_loss: 0.012167 | psnr: 19.148046 | perceptual_loss: 0.351673 | norm_perceptual_loss: 28.903042 [17:47:41.308035] [Epoch 2] | Iter: 43500 | Optim: 43500 | Time: 1.08s | LR: 0.000107 | SkipOpt: 0 loss: 0.196117 | l2_loss: 0.013362 | psnr: 18.741125 | perceptual_loss: 0.365509 | norm_perceptual_loss: 27.353359 [17:48:42.591318] [Epoch 2] | Iter: 43550 | Optim: 43550 | Time: 0.75s | LR: 0.000107 | SkipOpt: 0 loss: 0.162617 | l2_loss: 0.008860 | psnr: 20.525871 | perceptual_loss: 0.307514 | norm_perceptual_loss: 34.709831 [17:49:45.397845] [Epoch 2] | Iter: 43600 | Optim: 43600 | Time: 0.76s | LR: 0.000107 | SkipOpt: 0 loss: 0.166590 | l2_loss: 0.008207 | psnr: 20.858004 | perceptual_loss: 0.316765 | norm_perceptual_loss: 38.595551 [17:50:46.479352] [Epoch 2] | Iter: 43650 | Optim: 43650 | Time: 1.07s | LR: 0.000106 | SkipOpt: 0 loss: 0.200941 | l2_loss: 0.013131 | psnr: 18.816895 | perceptual_loss: 0.375620 | norm_perceptual_loss: 28.604736 [17:51:46.885313] [Epoch 2] | Iter: 43700 | Optim: 43700 | Time: 1.07s | LR: 0.000106 | SkipOpt: 0 loss: 0.186375 | l2_loss: 0.012727 | psnr: 18.952885 | perceptual_loss: 0.347297 | norm_perceptual_loss: 27.289070 [17:52:47.752462] [Epoch 2] | Iter: 43750 | Optim: 43750 | Time: 1.91s | LR: 0.000105 | SkipOpt: 0 loss: 0.173906 | l2_loss: 0.010846 | psnr: 19.647490 | perceptual_loss: 0.326121 | norm_perceptual_loss: 30.069633 [17:53:45.398100] [Epoch 2] | Iter: 43800 | Optim: 43800 | Time: 1.07s | LR: 0.000105 | SkipOpt: 0 loss: 0.213540 | l2_loss: 0.017576 | psnr: 17.550680 | perceptual_loss: 0.391926 | norm_perceptual_loss: 22.298342 [17:54:45.542191] [Epoch 2] | Iter: 43850 | Optim: 43850 | Time: 1.90s | LR: 0.000104 | SkipOpt: 0 loss: 0.217921 | l2_loss: 0.016030 | psnr: 17.950771 | perceptual_loss: 0.403784 | norm_perceptual_loss: 25.189865 [17:55:47.942264] [Epoch 2] | Iter: 43900 | Optim: 43900 | Time: 1.07s | LR: 0.000104 | SkipOpt: 0 loss: 0.177111 | l2_loss: 0.012862 | psnr: 18.906883 | perceptual_loss: 0.328497 | norm_perceptual_loss: 25.539957 [17:56:55.318070] [Epoch 2] | Iter: 43950 | Optim: 43950 | Time: 1.91s | LR: 0.000103 | SkipOpt: 0 loss: 0.197672 | l2_loss: 0.013149 | psnr: 18.811132 | perceptual_loss: 0.369046 | norm_perceptual_loss: 28.066830 [17:57:52.248344] [Epoch 2] | Iter: 44000 | Optim: 44000 | Time: 0.75s | LR: 0.000103 | SkipOpt: 0 loss: 0.221959 | l2_loss: 0.013408 | psnr: 18.726225 | perceptual_loss: 0.417102 | norm_perceptual_loss: 31.107468 [17:57:57.970726] Saved checkpoint at iteration 44000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [17:57:57.971182] Saved latest checkpoint at iteration 44000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [17:58:57.668266] [Epoch 2] | Iter: 44050 | Optim: 44050 | Time: 0.75s | LR: 0.000103 | SkipOpt: 0 loss: 0.159113 | l2_loss: 0.007897 | psnr: 21.025400 | perceptual_loss: 0.302431 | norm_perceptual_loss: 38.297161 [17:59:57.458064] [Epoch 2] | Iter: 44100 | Optim: 44100 | Time: 1.91s | LR: 0.000102 | SkipOpt: 0 loss: 0.169286 | l2_loss: 0.010178 | psnr: 19.923223 | perceptual_loss: 0.318215 | norm_perceptual_loss: 31.263895 [18:00:54.483479] [Epoch 2] | Iter: 44150 | Optim: 44150 | Time: 1.07s | LR: 0.000102 | SkipOpt: 0 loss: 0.173975 | l2_loss: 0.010143 | psnr: 19.938354 | perceptual_loss: 0.327665 | norm_perceptual_loss: 32.304699 [18:01:55.955288] [Epoch 2] | Iter: 44200 | Optim: 44200 | Time: 0.85s | LR: 0.000101 | SkipOpt: 0 loss: 0.178833 | l2_loss: 0.007542 | psnr: 21.225410 | perceptual_loss: 0.342583 | norm_perceptual_loss: 45.426277 [18:02:55.966665] [Epoch 2] | Iter: 44250 | Optim: 44250 | Time: 0.75s | LR: 0.000101 | SkipOpt: 0 loss: 0.183465 | l2_loss: 0.012148 | psnr: 19.155087 | perceptual_loss: 0.342636 | norm_perceptual_loss: 28.205982 [18:03:58.397798] [Epoch 2] | Iter: 44300 | Optim: 44300 | Time: 1.08s | LR: 0.000100 | SkipOpt: 0 loss: 0.200466 | l2_loss: 0.009904 | psnr: 20.041849 | perceptual_loss: 0.381124 | norm_perceptual_loss: 38.481438 [18:04:59.424190] [Epoch 2] | Iter: 44350 | Optim: 44350 | Time: 1.07s | LR: 0.000100 | SkipOpt: 0 loss: 0.159261 | l2_loss: 0.008739 | psnr: 20.585196 | perceptual_loss: 0.301043 | norm_perceptual_loss: 34.446754 [18:06:02.090352] [Epoch 2] | Iter: 44400 | Optim: 44400 | Time: 1.91s | LR: 0.000099 | SkipOpt: 0 loss: 0.220743 | l2_loss: 0.016933 | psnr: 17.712639 | perceptual_loss: 0.407620 | norm_perceptual_loss: 24.072428 [18:07:06.607939] [Epoch 2] | Iter: 44450 | Optim: 44450 | Time: 1.91s | LR: 0.000099 | SkipOpt: 0 loss: 0.230695 | l2_loss: 0.018062 | psnr: 17.432346 | perceptual_loss: 0.425267 | norm_perceptual_loss: 23.544861 [18:08:03.383633] [Epoch 2] | Iter: 44500 | Optim: 44500 | Time: 0.76s | LR: 0.000099 | SkipOpt: 0 loss: 0.171289 | l2_loss: 0.011366 | psnr: 19.444016 | perceptual_loss: 0.319846 | norm_perceptual_loss: 28.141157 [18:09:10.152437] [Epoch 2] | Iter: 44550 | Optim: 44550 | Time: 1.91s | LR: 0.000098 | SkipOpt: 0 loss: 0.176506 | l2_loss: 0.011121 | psnr: 19.538649 | perceptual_loss: 0.330771 | norm_perceptual_loss: 29.743517 [18:10:08.191577] [Epoch 2] | Iter: 44600 | Optim: 44600 | Time: 0.75s | LR: 0.000098 | SkipOpt: 0 loss: 0.180913 | l2_loss: 0.011819 | psnr: 19.274376 | perceptual_loss: 0.338189 | norm_perceptual_loss: 28.615248 [18:11:11.488274] [Epoch 2] | Iter: 44650 | Optim: 44650 | Time: 1.91s | LR: 0.000097 | SkipOpt: 0 loss: 0.179707 | l2_loss: 0.010528 | psnr: 19.776634 | perceptual_loss: 0.338359 | norm_perceptual_loss: 32.139660 [18:12:15.151546] [Epoch 2] | Iter: 44700 | Optim: 44700 | Time: 0.75s | LR: 0.000097 | SkipOpt: 0 loss: 0.168755 | l2_loss: 0.010605 | psnr: 19.744755 | perceptual_loss: 0.316299 | norm_perceptual_loss: 29.824488 [18:13:15.668720] [Epoch 2] | Iter: 44750 | Optim: 44750 | Time: 1.08s | LR: 0.000096 | SkipOpt: 0 loss: 0.170179 | l2_loss: 0.009834 | psnr: 20.072832 | perceptual_loss: 0.320690 | norm_perceptual_loss: 32.611309 [18:14:12.612320] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [18:14:20.856554] [Epoch 2] | Iter: 44800 | Optim: 44800 | Time: 0.75s | LR: 0.000096 | SkipOpt: 0 loss: 0.167523 | l2_loss: 0.009210 | psnr: 20.357368 | perceptual_loss: 0.316626 | norm_perceptual_loss: 34.378227 [18:15:20.402905] [Epoch 2] | Iter: 44850 | Optim: 44850 | Time: 0.75s | LR: 0.000096 | SkipOpt: 0 loss: 0.214317 | l2_loss: 0.010433 | psnr: 19.816031 | perceptual_loss: 0.407768 | norm_perceptual_loss: 39.085571 [18:16:24.150355] [Epoch 2] | Iter: 44900 | Optim: 44900 | Time: 1.91s | LR: 0.000095 | SkipOpt: 0 loss: 0.199504 | l2_loss: 0.013507 | psnr: 18.694351 | perceptual_loss: 0.371993 | norm_perceptual_loss: 27.540377 [18:17:25.312367] [Epoch 2] | Iter: 44950 | Optim: 44950 | Time: 0.85s | LR: 0.000095 | SkipOpt: 0 loss: 0.137575 | l2_loss: 0.007680 | psnr: 21.146290 | perceptual_loss: 0.259790 | norm_perceptual_loss: 33.826073 [18:18:31.961603] [Epoch 2] | Iter: 45000 | Optim: 45000 | Time: 1.91s | LR: 0.000094 | SkipOpt: 0 loss: 0.217671 | l2_loss: 0.015789 | psnr: 18.016445 | perceptual_loss: 0.403763 | norm_perceptual_loss: 25.572403 [18:18:33.548532] Found 140 sequences [18:18:52.861866] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [18:18:53.936982] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [18:19:45.306445] Gathering scores from all processes... [18:19:45.307328] Padding local tensor of length 140 to max count: 140 [18:19:45.307910] Padding local tensor of length 140 to max count: 140 [18:19:45.308387] Padding local tensor of length 140 to max count: 140 [18:19:45.308685] Gathering scene names from all processes... [18:19:45.308811] Padding local tensor of length 140 on rank 0 to max count: 140 [18:19:45.309828] Evaluation metrics: [18:19:45.310046] psnr: 17.4176 [18:19:45.310149] ssim: 0.4929 [18:19:45.310232] lpips: 0.4911 [18:19:45.313629] Found 140 sequences [18:20:08.632389] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [18:20:09.483323] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [18:21:02.050635] Gathering scores from all processes... [18:21:02.051448] Padding local tensor of length 140 to max count: 140 [18:21:02.051977] Padding local tensor of length 140 to max count: 140 [18:21:02.052419] Padding local tensor of length 140 to max count: 140 [18:21:02.052680] Gathering scene names from all processes... [18:21:02.052791] Padding local tensor of length 140 on rank 0 to max count: 140 [18:21:02.053796] Evaluation metrics: [18:21:02.053975] psnr: 20.2030 [18:21:02.054056] ssim: 0.5845 [18:21:02.054130] lpips: 0.3814 [18:21:02.057820] Found 140 sequences [18:21:25.603929] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [18:21:26.467945] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [18:22:22.075237] Gathering scores from all processes... [18:22:22.076076] Padding local tensor of length 140 to max count: 140 [18:22:22.076764] Padding local tensor of length 140 to max count: 140 [18:22:22.077449] Padding local tensor of length 140 to max count: 140 [18:22:22.077783] Gathering scene names from all processes... [18:22:22.077938] Padding local tensor of length 140 on rank 0 to max count: 140 [18:22:22.078967] Evaluation metrics: [18:22:22.079173] psnr: 21.2375 [18:22:22.079294] ssim: 0.6189 [18:22:22.079402] lpips: 0.3448 [18:22:22.080989] [Eval][iter 45000] dl3dv_2view | psnr: 17.417568 | ssim: 0.492875 | lpips: 0.491123 || dl3dv_4view | psnr: 20.203033 | ssim: 0.584476 | lpips: 0.381447 || dl3dv_6view | psnr: 21.237524 | ssim: 0.618938 | lpips: 0.344819 [18:22:22.084260] [Eval][iter 45000] dl3dv_2view | psnr: 17.417568 | ssim: 0.492875 | lpips: 0.491123 || dl3dv_4view | psnr: 20.203033 | ssim: 0.584476 | lpips: 0.381447 || dl3dv_6view | psnr: 21.237524 | ssim: 0.618938 | lpips: 0.344819 [18:22:26.385608] Saved checkpoint at iteration 45000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [18:22:26.386025] Saved latest checkpoint at iteration 45000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [18:23:29.213932] [Epoch 2] | Iter: 45050 | Optim: 45050 | Time: 1.07s | LR: 0.000094 | SkipOpt: 0 loss: 0.183071 | l2_loss: 0.010230 | psnr: 19.901176 | perceptual_loss: 0.345683 | norm_perceptual_loss: 33.790543 [18:24:29.854894] [Epoch 2] | Iter: 45100 | Optim: 45100 | Time: 1.90s | LR: 0.000093 | SkipOpt: 0 loss: 0.202558 | l2_loss: 0.019434 | psnr: 17.114355 | perceptual_loss: 0.366248 | norm_perceptual_loss: 18.845615 [18:25:34.452187] [Epoch 2] | Iter: 45150 | Optim: 45150 | Time: 1.91s | LR: 0.000093 | SkipOpt: 0 loss: 0.206718 | l2_loss: 0.014246 | psnr: 18.463045 | perceptual_loss: 0.384945 | norm_perceptual_loss: 27.021095 [18:26:32.790204] [Epoch 2] | Iter: 45200 | Optim: 45200 | Time: 0.75s | LR: 0.000092 | SkipOpt: 0 loss: 0.164774 | l2_loss: 0.008500 | psnr: 20.705683 | perceptual_loss: 0.312548 | norm_perceptual_loss: 36.769238 [18:27:34.624344] [Epoch 2] | Iter: 45250 | Optim: 45250 | Time: 0.75s | LR: 0.000092 | SkipOpt: 0 loss: 0.167373 | l2_loss: 0.011178 | psnr: 19.516420 | perceptual_loss: 0.312390 | norm_perceptual_loss: 27.947252 [18:28:35.923435] [Epoch 2] | Iter: 45300 | Optim: 45300 | Time: 0.76s | LR: 0.000092 | SkipOpt: 0 loss: 0.179451 | l2_loss: 0.008367 | psnr: 20.774368 | perceptual_loss: 0.342168 | norm_perceptual_loss: 40.895599 [18:29:37.969887] [Epoch 2] | Iter: 45350 | Optim: 45350 | Time: 0.75s | LR: 0.000091 | SkipOpt: 0 loss: 0.146738 | l2_loss: 0.006959 | psnr: 21.574553 | perceptual_loss: 0.279559 | norm_perceptual_loss: 40.172474 [18:30:34.646397] [Epoch 2] | Iter: 45400 | Optim: 45400 | Time: 1.27s | LR: 0.000091 | SkipOpt: 0 loss: 0.190272 | l2_loss: 0.014905 | psnr: 18.266552 | perceptual_loss: 0.350733 | norm_perceptual_loss: 23.530523 [18:31:41.899864] [Epoch 2] | Iter: 45450 | Optim: 45450 | Time: 1.92s | LR: 0.000090 | SkipOpt: 0 loss: 0.193332 | l2_loss: 0.013774 | psnr: 18.609533 | perceptual_loss: 0.359116 | norm_perceptual_loss: 26.072821 [18:32:48.745395] [Epoch 2] | Iter: 45500 | Optim: 45500 | Time: 0.75s | LR: 0.000090 | SkipOpt: 0 loss: 0.147528 | l2_loss: 0.012021 | psnr: 19.200762 | perceptual_loss: 0.271015 | norm_perceptual_loss: 22.545980 [18:33:49.195872] [Epoch 2] | Iter: 45550 | Optim: 45550 | Time: 0.75s | LR: 0.000090 | SkipOpt: 0 loss: 0.168321 | l2_loss: 0.008665 | psnr: 20.622456 | perceptual_loss: 0.319313 | norm_perceptual_loss: 36.852058 [18:34:46.829455] [Epoch 2] | Iter: 45600 | Optim: 45600 | Time: 0.75s | LR: 0.000089 | SkipOpt: 0 loss: 0.168738 | l2_loss: 0.007859 | psnr: 21.046400 | perceptual_loss: 0.321758 | norm_perceptual_loss: 40.941978 [18:35:44.614181] [Epoch 2] | Iter: 45650 | Optim: 45650 | Time: 1.91s | LR: 0.000089 | SkipOpt: 0 loss: 0.223512 | l2_loss: 0.017706 | psnr: 17.518890 | perceptual_loss: 0.411613 | norm_perceptual_loss: 23.247606 [18:36:50.199390] [Epoch 2] | Iter: 45700 | Optim: 45700 | Time: 1.92s | LR: 0.000088 | SkipOpt: 0 loss: 0.212081 | l2_loss: 0.014646 | psnr: 18.342941 | perceptual_loss: 0.394870 | norm_perceptual_loss: 26.961792 [18:37:54.513228] [Epoch 2] | Iter: 45750 | Optim: 45750 | Time: 0.86s | LR: 0.000088 | SkipOpt: 0 loss: 0.171267 | l2_loss: 0.008144 | psnr: 20.891399 | perceptual_loss: 0.326246 | norm_perceptual_loss: 40.057613 [18:38:57.986771] [Epoch 2] | Iter: 45800 | Optim: 45800 | Time: 0.85s | LR: 0.000087 | SkipOpt: 0 loss: 0.168058 | l2_loss: 0.007186 | psnr: 21.434841 | perceptual_loss: 0.321744 | norm_perceptual_loss: 44.770706 [18:39:59.207126] [Epoch 2] | Iter: 45850 | Optim: 45850 | Time: 1.07s | LR: 0.000087 | SkipOpt: 0 loss: 0.212748 | l2_loss: 0.013734 | psnr: 18.622105 | perceptual_loss: 0.398029 | norm_perceptual_loss: 28.981802 [18:40:57.813422] [Epoch 2] | Iter: 45900 | Optim: 45900 | Time: 0.75s | LR: 0.000087 | SkipOpt: 0 loss: 0.151655 | l2_loss: 0.006769 | psnr: 21.694622 | perceptual_loss: 0.289771 | norm_perceptual_loss: 42.807251 [18:41:59.363836] [Epoch 2] | Iter: 45950 | Optim: 45950 | Time: 1.08s | LR: 0.000086 | SkipOpt: 0 loss: 0.144506 | l2_loss: 0.008181 | psnr: 20.872114 | perceptual_loss: 0.272651 | norm_perceptual_loss: 33.328686 [18:43:03.094826] [Epoch 2] | Iter: 46000 | Optim: 46000 | Time: 0.76s | LR: 0.000086 | SkipOpt: 0 loss: 0.152803 | l2_loss: 0.007528 | psnr: 21.233204 | perceptual_loss: 0.290549 | norm_perceptual_loss: 38.595814 [18:43:08.435972] Saved checkpoint at iteration 46000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [18:43:08.436321] Saved latest checkpoint at iteration 46000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [18:44:05.493570] [Epoch 2] | Iter: 46050 | Optim: 46050 | Time: 1.07s | LR: 0.000085 | SkipOpt: 0 loss: 0.178464 | l2_loss: 0.011554 | psnr: 19.372801 | perceptual_loss: 0.333820 | norm_perceptual_loss: 28.892948 [18:45:05.133535] [Epoch 2] | Iter: 46100 | Optim: 46100 | Time: 1.07s | LR: 0.000085 | SkipOpt: 0 loss: 0.150354 | l2_loss: 0.006891 | psnr: 21.616867 | perceptual_loss: 0.286926 | norm_perceptual_loss: 41.634792 [18:46:08.222100] [Epoch 2] | Iter: 46150 | Optim: 46150 | Time: 0.75s | LR: 0.000085 | SkipOpt: 0 loss: 0.165283 | l2_loss: 0.010570 | psnr: 19.759270 | perceptual_loss: 0.309427 | norm_perceptual_loss: 29.274187 [18:47:09.329419] [Epoch 2] | Iter: 46200 | Optim: 46200 | Time: 1.92s | LR: 0.000084 | SkipOpt: 0 loss: 0.187297 | l2_loss: 0.012497 | psnr: 19.032042 | perceptual_loss: 0.349601 | norm_perceptual_loss: 27.975395 [18:48:08.581173] [Epoch 2] | Iter: 46250 | Optim: 46250 | Time: 0.76s | LR: 0.000084 | SkipOpt: 0 loss: 0.134134 | l2_loss: 0.005012 | psnr: 22.999950 | perceptual_loss: 0.258244 | norm_perceptual_loss: 51.525814 [18:49:12.319284] [Epoch 2] | Iter: 46300 | Optim: 46300 | Time: 1.92s | LR: 0.000083 | SkipOpt: 0 loss: 0.190349 | l2_loss: 0.012693 | psnr: 18.964420 | perceptual_loss: 0.355313 | norm_perceptual_loss: 27.993221 [18:50:18.272398] [Epoch 2] | Iter: 46350 | Optim: 46350 | Time: 1.91s | LR: 0.000083 | SkipOpt: 0 loss: 0.225986 | l2_loss: 0.018643 | psnr: 17.294819 | perceptual_loss: 0.414687 | norm_perceptual_loss: 22.243433 [18:51:21.142740] [Epoch 2] | Iter: 46400 | Optim: 46400 | Time: 1.07s | LR: 0.000082 | SkipOpt: 0 loss: 0.186771 | l2_loss: 0.011408 | psnr: 19.427858 | perceptual_loss: 0.350727 | norm_perceptual_loss: 30.743599 [18:52:23.932174] [Epoch 2] | Iter: 46450 | Optim: 46450 | Time: 0.85s | LR: 0.000082 | SkipOpt: 0 loss: 0.152508 | l2_loss: 0.009642 | psnr: 20.158178 | perceptual_loss: 0.285732 | norm_perceptual_loss: 29.633057 [18:53:26.256523] [Epoch 2] | Iter: 46500 | Optim: 46500 | Time: 1.91s | LR: 0.000082 | SkipOpt: 0 loss: 0.195545 | l2_loss: 0.014504 | psnr: 18.385195 | perceptual_loss: 0.362083 | norm_perceptual_loss: 24.964777 [18:54:28.924363] [Epoch 2] | Iter: 46550 | Optim: 46550 | Time: 0.75s | LR: 0.000081 | SkipOpt: 0 loss: 0.163953 | l2_loss: 0.012603 | psnr: 18.995169 | perceptual_loss: 0.302699 | norm_perceptual_loss: 24.017481 [18:55:34.421357] [Epoch 2] | Iter: 46600 | Optim: 46600 | Time: 1.91s | LR: 0.000081 | SkipOpt: 0 loss: 0.192009 | l2_loss: 0.016503 | psnr: 17.824409 | perceptual_loss: 0.351012 | norm_perceptual_loss: 21.269747 [18:56:36.394595] [Epoch 2] | Iter: 46650 | Optim: 46650 | Time: 0.85s | LR: 0.000080 | SkipOpt: 0 loss: 0.214769 | l2_loss: 0.015036 | psnr: 18.228622 | perceptual_loss: 0.399466 | norm_perceptual_loss: 26.566965 [18:57:43.394499] [Epoch 2] | Iter: 46700 | Optim: 46700 | Time: 1.91s | LR: 0.000080 | SkipOpt: 0 loss: 0.205663 | l2_loss: 0.016005 | psnr: 17.957483 | perceptual_loss: 0.379316 | norm_perceptual_loss: 23.700060 [18:58:47.043282] [Epoch 2] | Iter: 46750 | Optim: 46750 | Time: 1.91s | LR: 0.000080 | SkipOpt: 0 loss: 0.193203 | l2_loss: 0.014204 | psnr: 18.476046 | perceptual_loss: 0.358000 | norm_perceptual_loss: 25.205040 [18:59:49.904243] [Epoch 2] | Iter: 46800 | Optim: 46800 | Time: 1.91s | LR: 0.000079 | SkipOpt: 0 loss: 0.199341 | l2_loss: 0.014114 | psnr: 18.503395 | perceptual_loss: 0.370453 | norm_perceptual_loss: 26.246563 [19:00:49.890840] [Epoch 2] | Iter: 46850 | Optim: 46850 | Time: 0.75s | LR: 0.000079 | SkipOpt: 0 loss: 0.210385 | l2_loss: 0.010425 | psnr: 19.819138 | perceptual_loss: 0.399919 | norm_perceptual_loss: 38.360661 [19:01:53.856777] [Epoch 2] | Iter: 46900 | Optim: 46900 | Time: 1.08s | LR: 0.000078 | SkipOpt: 0 loss: 0.169187 | l2_loss: 0.009710 | psnr: 20.127850 | perceptual_loss: 0.318955 | norm_perceptual_loss: 32.848366 [19:02:50.650709] [Epoch 2] | Iter: 46950 | Optim: 46950 | Time: 0.85s | LR: 0.000078 | SkipOpt: 0 loss: 0.168618 | l2_loss: 0.010135 | psnr: 19.941957 | perceptual_loss: 0.316968 | norm_perceptual_loss: 31.275961 [19:03:56.238658] [Epoch 2] | Iter: 47000 | Optim: 47000 | Time: 1.08s | LR: 0.000078 | SkipOpt: 0 loss: 0.197512 | l2_loss: 0.012495 | psnr: 19.032801 | perceptual_loss: 0.370035 | norm_perceptual_loss: 29.615770 [19:04:01.943912] Saved checkpoint at iteration 47000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [19:04:01.944234] Saved latest checkpoint at iteration 47000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [19:05:07.811069] [Epoch 2] | Iter: 47050 | Optim: 47050 | Time: 0.75s | LR: 0.000077 | SkipOpt: 0 loss: 0.135316 | l2_loss: 0.008802 | psnr: 20.554335 | perceptual_loss: 0.253029 | norm_perceptual_loss: 28.747705 [19:06:12.228598] [Epoch 2] | Iter: 47100 | Optim: 47100 | Time: 1.91s | LR: 0.000077 | SkipOpt: 0 loss: 0.232197 | l2_loss: 0.020170 | psnr: 16.952900 | perceptual_loss: 0.424054 | norm_perceptual_loss: 21.023811 [19:07:11.708923] [Epoch 2] | Iter: 47150 | Optim: 47150 | Time: 0.75s | LR: 0.000076 | SkipOpt: 0 loss: 0.180656 | l2_loss: 0.008640 | psnr: 20.634699 | perceptual_loss: 0.344032 | norm_perceptual_loss: 39.817047 [19:08:12.534378] [Epoch 2] | Iter: 47200 | Optim: 47200 | Time: 0.85s | LR: 0.000076 | SkipOpt: 0 loss: 0.196901 | l2_loss: 0.013443 | psnr: 18.715145 | perceptual_loss: 0.366917 | norm_perceptual_loss: 27.294937 [19:09:16.423805] [Epoch 2] | Iter: 47250 | Optim: 47250 | Time: 1.08s | LR: 0.000076 | SkipOpt: 0 loss: 0.192328 | l2_loss: 0.012848 | psnr: 18.911646 | perceptual_loss: 0.358960 | norm_perceptual_loss: 27.939003 [19:10:19.074344] [Epoch 2] | Iter: 47300 | Optim: 47300 | Time: 1.08s | LR: 0.000075 | SkipOpt: 0 loss: 0.155599 | l2_loss: 0.008271 | psnr: 20.824463 | perceptual_loss: 0.294656 | norm_perceptual_loss: 35.625519 [19:11:21.843624] [Epoch 2] | Iter: 47350 | Optim: 47350 | Time: 0.76s | LR: 0.000075 | SkipOpt: 0 loss: 0.159371 | l2_loss: 0.007163 | psnr: 21.449125 | perceptual_loss: 0.304416 | norm_perceptual_loss: 42.499092 [19:12:28.648850] [Epoch 2] | Iter: 47400 | Optim: 47400 | Time: 3.71s | LR: 0.000074 | SkipOpt: 0 loss: 0.211787 | l2_loss: 0.016504 | psnr: 17.824026 | perceptual_loss: 0.390565 | norm_perceptual_loss: 23.664402 [19:13:30.924924] [Epoch 2] | Iter: 47450 | Optim: 47450 | Time: 0.75s | LR: 0.000074 | SkipOpt: 0 loss: 0.145108 | l2_loss: 0.006967 | psnr: 21.569592 | perceptual_loss: 0.276283 | norm_perceptual_loss: 39.656414 [19:14:29.110335] [Epoch 2] | Iter: 47500 | Optim: 47500 | Time: 1.92s | LR: 0.000074 | SkipOpt: 0 loss: 0.177975 | l2_loss: 0.013277 | psnr: 18.769073 | perceptual_loss: 0.329396 | norm_perceptual_loss: 24.809933 [19:15:31.068253] [Epoch 2] | Iter: 47550 | Optim: 47550 | Time: 0.75s | LR: 0.000073 | SkipOpt: 0 loss: 0.145165 | l2_loss: 0.008141 | psnr: 20.893476 | perceptual_loss: 0.274049 | norm_perceptual_loss: 33.664764 [19:16:37.212063] [Epoch 2] | Iter: 47600 | Optim: 47600 | Time: 1.07s | LR: 0.000073 | SkipOpt: 0 loss: 0.150338 | l2_loss: 0.009263 | psnr: 20.332687 | perceptual_loss: 0.282152 | norm_perceptual_loss: 30.461494 [19:17:38.490675] [Epoch 2] | Iter: 47650 | Optim: 47650 | Time: 1.07s | LR: 0.000072 | SkipOpt: 0 loss: 0.199192 | l2_loss: 0.012912 | psnr: 18.890217 | perceptual_loss: 0.372561 | norm_perceptual_loss: 28.854897 [19:18:37.357195] [Epoch 2] | Iter: 47700 | Optim: 47700 | Time: 0.85s | LR: 0.000072 | SkipOpt: 0 loss: 0.179072 | l2_loss: 0.011402 | psnr: 19.430058 | perceptual_loss: 0.335340 | norm_perceptual_loss: 29.409758 [19:19:40.190922] [Epoch 2] | Iter: 47750 | Optim: 47750 | Time: 1.08s | LR: 0.000072 | SkipOpt: 0 loss: 0.181508 | l2_loss: 0.013513 | psnr: 18.692417 | perceptual_loss: 0.335989 | norm_perceptual_loss: 24.863758 [19:20:36.414472] [Epoch 2] | Iter: 47800 | Optim: 47800 | Time: 1.07s | LR: 0.000071 | SkipOpt: 0 loss: 0.210956 | l2_loss: 0.015273 | psnr: 18.160786 | perceptual_loss: 0.391367 | norm_perceptual_loss: 25.624943 [19:21:37.230312] [Epoch 2] | Iter: 47850 | Optim: 47850 | Time: 1.08s | LR: 0.000071 | SkipOpt: 0 loss: 0.162869 | l2_loss: 0.009618 | psnr: 20.168941 | perceptual_loss: 0.306502 | norm_perceptual_loss: 31.865955 [19:22:38.925138] [Epoch 2] | Iter: 47900 | Optim: 47900 | Time: 1.08s | LR: 0.000071 | SkipOpt: 0 loss: 0.176051 | l2_loss: 0.012415 | psnr: 19.060581 | perceptual_loss: 0.327271 | norm_perceptual_loss: 26.361259 [19:23:41.933908] [Epoch 2] | Iter: 47950 | Optim: 47950 | Time: 1.07s | LR: 0.000070 | SkipOpt: 0 loss: 0.206167 | l2_loss: 0.013430 | psnr: 18.719202 | perceptual_loss: 0.385473 | norm_perceptual_loss: 28.702148 [19:24:45.259938] [Epoch 2] | Iter: 48000 | Optim: 48000 | Time: 1.91s | LR: 0.000070 | SkipOpt: 0 loss: 0.188163 | l2_loss: 0.013311 | psnr: 18.757820 | perceptual_loss: 0.349705 | norm_perceptual_loss: 26.271397 [19:24:50.660036] Saved checkpoint at iteration 48000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [19:24:50.660478] Saved latest checkpoint at iteration 48000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [19:25:51.847549] [Epoch 2] | Iter: 48050 | Optim: 48050 | Time: 1.91s | LR: 0.000069 | SkipOpt: 0 loss: 0.221919 | l2_loss: 0.020270 | psnr: 16.931404 | perceptual_loss: 0.403297 | norm_perceptual_loss: 19.895969 [19:26:55.707228] [Epoch 2] | Iter: 48100 | Optim: 48100 | Time: 1.27s | LR: 0.000069 | SkipOpt: 0 loss: 0.164954 | l2_loss: 0.009489 | psnr: 20.227917 | perceptual_loss: 0.310930 | norm_perceptual_loss: 32.768337 [19:27:49.836442] [Epoch 2] | Iter: 48150 | Optim: 48150 | Time: 0.75s | LR: 0.000069 | SkipOpt: 0 loss: 0.159361 | l2_loss: 0.006508 | psnr: 21.865719 | perceptual_loss: 0.305707 | norm_perceptual_loss: 46.976173 [19:28:50.193528] [Epoch 2] | Iter: 48200 | Optim: 48200 | Time: 0.75s | LR: 0.000068 | SkipOpt: 0 loss: 0.175938 | l2_loss: 0.009501 | psnr: 20.222292 | perceptual_loss: 0.332874 | norm_perceptual_loss: 35.035519 [19:29:46.698165] [Epoch 2] | Iter: 48250 | Optim: 48250 | Time: 1.07s | LR: 0.000068 | SkipOpt: 0 loss: 0.189078 | l2_loss: 0.010277 | psnr: 19.881269 | perceptual_loss: 0.357602 | norm_perceptual_loss: 34.795803 [19:30:45.229933] [Epoch 2] | Iter: 48300 | Optim: 48300 | Time: 1.91s | LR: 0.000067 | SkipOpt: 0 loss: 0.192231 | l2_loss: 0.012306 | psnr: 19.098961 | perceptual_loss: 0.359851 | norm_perceptual_loss: 29.242758 [19:31:44.435679] [Epoch 2] | Iter: 48350 | Optim: 48350 | Time: 1.08s | LR: 0.000067 | SkipOpt: 0 loss: 0.188060 | l2_loss: 0.012516 | psnr: 19.025242 | perceptual_loss: 0.351088 | norm_perceptual_loss: 28.050491 [19:32:57.626593] [Epoch 2] | Iter: 48400 | Optim: 48400 | Time: 0.75s | LR: 0.000067 | SkipOpt: 0 loss: 0.173810 | l2_loss: 0.009641 | psnr: 20.158751 | perceptual_loss: 0.328337 | norm_perceptual_loss: 34.056129 [19:34:08.791476] [Epoch 2] | Iter: 48450 | Optim: 48450 | Time: 1.91s | LR: 0.000066 | SkipOpt: 0 loss: 0.193273 | l2_loss: 0.014095 | psnr: 18.509501 | perceptual_loss: 0.358358 | norm_perceptual_loss: 25.425358 [19:35:14.595111] [Epoch 2] | Iter: 48500 | Optim: 48500 | Time: 0.75s | LR: 0.000066 | SkipOpt: 0 loss: 0.133870 | l2_loss: 0.005241 | psnr: 22.806177 | perceptual_loss: 0.257259 | norm_perceptual_loss: 49.089409 [19:36:19.083882] [Epoch 2] | Iter: 48550 | Optim: 48550 | Time: 1.07s | LR: 0.000066 | SkipOpt: 0 loss: 0.161267 | l2_loss: 0.008053 | psnr: 20.940292 | perceptual_loss: 0.306427 | norm_perceptual_loss: 38.050186 [19:37:24.286863] [Epoch 2] | Iter: 48600 | Optim: 48600 | Time: 0.75s | LR: 0.000065 | SkipOpt: 0 loss: 0.147761 | l2_loss: 0.006233 | psnr: 22.052895 | perceptual_loss: 0.283055 | norm_perceptual_loss: 45.410957 [19:38:23.994676] [Epoch 2] | Iter: 48650 | Optim: 48650 | Time: 0.85s | LR: 0.000065 | SkipOpt: 0 loss: 0.167917 | l2_loss: 0.007909 | psnr: 21.019016 | perceptual_loss: 0.320016 | norm_perceptual_loss: 40.464420 [19:39:18.072815] [Epoch 2] | Iter: 48700 | Optim: 48700 | Time: 1.91s | LR: 0.000064 | SkipOpt: 0 loss: 0.184368 | l2_loss: 0.012287 | psnr: 19.105438 | perceptual_loss: 0.344160 | norm_perceptual_loss: 28.009466 [19:40:24.627932] [Epoch 2] | Iter: 48750 | Optim: 48750 | Time: 1.91s | LR: 0.000064 | SkipOpt: 0 loss: 0.173937 | l2_loss: 0.012347 | psnr: 19.084513 | perceptual_loss: 0.323180 | norm_perceptual_loss: 26.175524 [19:41:20.826459] [Epoch 2] | Iter: 48800 | Optim: 48800 | Time: 0.75s | LR: 0.000064 | SkipOpt: 0 loss: 0.165210 | l2_loss: 0.008491 | psnr: 20.710304 | perceptual_loss: 0.313438 | norm_perceptual_loss: 36.913197 [19:42:16.414964] [Epoch 2] | Iter: 48850 | Optim: 48850 | Time: 1.91s | LR: 0.000063 | SkipOpt: 0 loss: 0.223112 | l2_loss: 0.020088 | psnr: 16.970575 | perceptual_loss: 0.406048 | norm_perceptual_loss: 20.213184 [19:43:15.088492] [Epoch 2] | Iter: 48900 | Optim: 48900 | Time: 1.27s | LR: 0.000063 | SkipOpt: 0 loss: 0.204532 | l2_loss: 0.015069 | psnr: 18.219069 | perceptual_loss: 0.378925 | norm_perceptual_loss: 25.145470 [19:44:16.933441] [Epoch 2] | Iter: 48950 | Optim: 48950 | Time: 0.75s | LR: 0.000063 | SkipOpt: 0 loss: 0.162405 | l2_loss: 0.010831 | psnr: 19.653385 | perceptual_loss: 0.303148 | norm_perceptual_loss: 27.989380 [19:45:21.493809] [Epoch 2] | Iter: 49000 | Optim: 49000 | Time: 1.91s | LR: 0.000062 | SkipOpt: 0 loss: 0.217826 | l2_loss: 0.015704 | psnr: 18.040007 | perceptual_loss: 0.404245 | norm_perceptual_loss: 25.742180 [19:45:26.937128] Saved checkpoint at iteration 49000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [19:45:26.937560] Saved latest checkpoint at iteration 49000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [19:46:35.006892] [Epoch 2] | Iter: 49050 | Optim: 49050 | Time: 1.08s | LR: 0.000062 | SkipOpt: 0 loss: 0.222142 | l2_loss: 0.015473 | psnr: 18.104275 | perceptual_loss: 0.413338 | norm_perceptual_loss: 26.713634 [19:47:38.718876] [Epoch 2] | Iter: 49100 | Optim: 49100 | Time: 1.91s | LR: 0.000061 | SkipOpt: 0 loss: 0.200820 | l2_loss: 0.013991 | psnr: 18.541431 | perceptual_loss: 0.373658 | norm_perceptual_loss: 26.706524 [19:48:41.172154] [Epoch 2] | Iter: 49150 | Optim: 49150 | Time: 0.75s | LR: 0.000061 | SkipOpt: 0 loss: 0.138687 | l2_loss: 0.006000 | psnr: 22.218332 | perceptual_loss: 0.265373 | norm_perceptual_loss: 44.227238 [19:49:38.566959] [Epoch 2] | Iter: 49200 | Optim: 49200 | Time: 1.08s | LR: 0.000061 | SkipOpt: 0 loss: 0.167930 | l2_loss: 0.008792 | psnr: 20.559370 | perceptual_loss: 0.318277 | norm_perceptual_loss: 36.202801 [19:50:39.289954] [Epoch 2] | Iter: 49250 | Optim: 49250 | Time: 1.91s | LR: 0.000060 | SkipOpt: 0 loss: 0.225790 | l2_loss: 0.016278 | psnr: 17.883970 | perceptual_loss: 0.419023 | norm_perceptual_loss: 25.741570 [19:51:37.133766] [Epoch 2] | Iter: 49300 | Optim: 49300 | Time: 0.75s | LR: 0.000060 | SkipOpt: 0 loss: 0.196001 | l2_loss: 0.012067 | psnr: 19.183851 | perceptual_loss: 0.367867 | norm_perceptual_loss: 30.484255 [19:52:36.761278] [Epoch 2] | Iter: 49350 | Optim: 49350 | Time: 0.76s | LR: 0.000060 | SkipOpt: 0 loss: 0.193977 | l2_loss: 0.012471 | psnr: 19.041145 | perceptual_loss: 0.363013 | norm_perceptual_loss: 29.109642 [19:53:37.872144] [Epoch 2] | Iter: 49400 | Optim: 49400 | Time: 1.07s | LR: 0.000059 | SkipOpt: 0 loss: 0.181037 | l2_loss: 0.012105 | psnr: 19.170221 | perceptual_loss: 0.337863 | norm_perceptual_loss: 27.910219 [19:54:37.747644] [Epoch 2] | Iter: 49450 | Optim: 49450 | Time: 1.27s | LR: 0.000059 | SkipOpt: 0 loss: 0.216903 | l2_loss: 0.014877 | psnr: 18.274948 | perceptual_loss: 0.404052 | norm_perceptual_loss: 27.160166 [19:55:13.196509] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [19:55:32.732938] [Epoch 2] | Iter: 49500 | Optim: 49500 | Time: 1.08s | LR: 0.000059 | SkipOpt: 0 loss: 0.179892 | l2_loss: 0.009796 | psnr: 20.089684 | perceptual_loss: 0.340193 | norm_perceptual_loss: 34.729156 [19:56:33.784425] [Epoch 2] | Iter: 49550 | Optim: 49550 | Time: 1.91s | LR: 0.000058 | SkipOpt: 0 loss: 0.182658 | l2_loss: 0.013635 | psnr: 18.653585 | perceptual_loss: 0.338046 | norm_perceptual_loss: 24.793312 [19:57:40.746326] [Epoch 2] | Iter: 49600 | Optim: 49600 | Time: 0.75s | LR: 0.000058 | SkipOpt: 0 loss: 0.184754 | l2_loss: 0.011702 | psnr: 19.317308 | perceptual_loss: 0.346103 | norm_perceptual_loss: 29.575802 [19:58:43.817321] [Epoch 2] | Iter: 49650 | Optim: 49650 | Time: 1.07s | LR: 0.000058 | SkipOpt: 0 loss: 0.193147 | l2_loss: 0.011705 | psnr: 19.316168 | perceptual_loss: 0.362884 | norm_perceptual_loss: 31.001612 [19:59:43.476856] [Epoch 2] | Iter: 49700 | Optim: 49700 | Time: 1.07s | LR: 0.000057 | SkipOpt: 0 loss: 0.154282 | l2_loss: 0.007907 | psnr: 21.019871 | perceptual_loss: 0.292750 | norm_perceptual_loss: 37.024101 [20:00:45.918686] [Epoch 2] | Iter: 49750 | Optim: 49750 | Time: 1.91s | LR: 0.000057 | SkipOpt: 0 loss: 0.212431 | l2_loss: 0.017224 | psnr: 17.638620 | perceptual_loss: 0.390414 | norm_perceptual_loss: 22.666677 [20:01:49.941220] [Epoch 2] | Iter: 49800 | Optim: 49800 | Time: 1.91s | LR: 0.000056 | SkipOpt: 0 loss: 0.226234 | l2_loss: 0.018839 | psnr: 17.249487 | perceptual_loss: 0.414790 | norm_perceptual_loss: 22.017965 [20:02:51.703662] [Epoch 2] | Iter: 49850 | Optim: 49850 | Time: 1.91s | LR: 0.000056 | SkipOpt: 0 loss: 0.256896 | l2_loss: 0.021103 | psnr: 16.756552 | perceptual_loss: 0.471585 | norm_perceptual_loss: 22.346790 [20:03:45.702843] [Epoch 2] | Iter: 49900 | Optim: 49900 | Time: 0.75s | LR: 0.000056 | SkipOpt: 0 loss: 0.165506 | l2_loss: 0.010673 | psnr: 19.717115 | perceptual_loss: 0.309666 | norm_perceptual_loss: 29.013798 [20:04:45.261058] [Epoch 2] | Iter: 49950 | Optim: 49950 | Time: 0.75s | LR: 0.000055 | SkipOpt: 0 loss: 0.164558 | l2_loss: 0.009049 | psnr: 20.434040 | perceptual_loss: 0.311018 | norm_perceptual_loss: 34.370785 [20:05:47.864421] [Epoch 2] | Iter: 50000 | Optim: 50000 | Time: 1.91s | LR: 0.000055 | SkipOpt: 0 loss: 0.184918 | l2_loss: 0.014259 | psnr: 18.459141 | perceptual_loss: 0.341318 | norm_perceptual_loss: 23.937178 [20:05:49.421932] Found 140 sequences [20:06:12.737702] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [20:06:13.773675] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [20:07:06.488230] Gathering scores from all processes... [20:07:06.489132] Padding local tensor of length 140 to max count: 140 [20:07:06.489663] Padding local tensor of length 140 to max count: 140 [20:07:06.490112] Padding local tensor of length 140 to max count: 140 [20:07:06.490380] Gathering scene names from all processes... [20:07:06.490489] Padding local tensor of length 140 on rank 0 to max count: 140 [20:07:06.491526] Evaluation metrics: [20:07:06.491697] psnr: 17.4758 [20:07:06.491778] ssim: 0.4967 [20:07:06.491852] lpips: 0.4799 [20:07:06.495551] Found 140 sequences [20:07:30.042744] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [20:07:31.054659] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [20:08:25.292726] Gathering scores from all processes... [20:08:25.293554] Padding local tensor of length 140 to max count: 140 [20:08:25.294114] Padding local tensor of length 140 to max count: 140 [20:08:25.294602] Padding local tensor of length 140 to max count: 140 [20:08:25.294906] Gathering scene names from all processes... [20:08:25.295046] Padding local tensor of length 140 on rank 0 to max count: 140 [20:08:25.296074] Evaluation metrics: [20:08:25.296282] psnr: 20.3868 [20:08:25.296385] ssim: 0.5944 [20:08:25.296485] lpips: 0.3696 [20:08:25.300059] Found 140 sequences [20:08:48.867375] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [20:08:49.742216] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [20:09:46.712195] Gathering scores from all processes... [20:09:46.713244] Padding local tensor of length 140 to max count: 140 [20:09:46.714108] Padding local tensor of length 140 to max count: 140 [20:09:46.715167] Padding local tensor of length 140 to max count: 140 [20:09:46.715703] Gathering scene names from all processes... [20:09:46.715845] Padding local tensor of length 140 on rank 0 to max count: 140 [20:09:46.717594] Evaluation metrics: [20:09:46.717978] psnr: 21.4833 [20:09:46.718085] ssim: 0.6315 [20:09:46.718230] lpips: 0.3318 [20:09:46.720906] [Eval][iter 50000] dl3dv_2view | psnr: 17.475805 | ssim: 0.496668 | lpips: 0.479948 || dl3dv_4view | psnr: 20.386845 | ssim: 0.594407 | lpips: 0.369613 || dl3dv_6view | psnr: 21.483255 | ssim: 0.631511 | lpips: 0.331847 [20:09:46.725290] [Eval][iter 50000] dl3dv_2view | psnr: 17.475805 | ssim: 0.496668 | lpips: 0.479948 || dl3dv_4view | psnr: 20.386845 | ssim: 0.594407 | lpips: 0.369613 || dl3dv_6view | psnr: 21.483255 | ssim: 0.631511 | lpips: 0.331847 [20:09:52.518626] Saved checkpoint at iteration 50000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [20:09:52.519422] Saved latest checkpoint at iteration 50000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [20:10:52.774835] [Epoch 2] | Iter: 50050 | Optim: 50050 | Time: 1.07s | LR: 0.000055 | SkipOpt: 0 loss: 0.168508 | l2_loss: 0.012216 | psnr: 19.130539 | perceptual_loss: 0.312583 | norm_perceptual_loss: 25.587027 [20:11:53.038494] [Epoch 2] | Iter: 50100 | Optim: 50100 | Time: 1.07s | LR: 0.000054 | SkipOpt: 0 loss: 0.162843 | l2_loss: 0.007088 | psnr: 21.494545 | perceptual_loss: 0.311509 | norm_perceptual_loss: 43.946560 [20:12:53.865605] [Epoch 2] | Iter: 50150 | Optim: 50150 | Time: 1.91s | LR: 0.000054 | SkipOpt: 0 loss: 0.190631 | l2_loss: 0.012490 | psnr: 19.034357 | perceptual_loss: 0.356282 | norm_perceptual_loss: 28.525242 [20:13:58.786888] [Epoch 2] | Iter: 50200 | Optim: 50200 | Time: 1.91s | LR: 0.000054 | SkipOpt: 0 loss: 0.232589 | l2_loss: 0.017110 | psnr: 17.667551 | perceptual_loss: 0.430958 | norm_perceptual_loss: 25.187811 [20:15:02.690732] [Epoch 2] | Iter: 50250 | Optim: 50250 | Time: 0.80s | LR: 0.000053 | SkipOpt: 0 loss: 0.162617 | l2_loss: 0.007715 | psnr: 21.126629 | perceptual_loss: 0.309804 | norm_perceptual_loss: 40.155964 [20:16:08.783159] [Epoch 2] | Iter: 50300 | Optim: 50300 | Time: 0.78s | LR: 0.000053 | SkipOpt: 0 loss: 0.171821 | l2_loss: 0.010201 | psnr: 19.913723 | perceptual_loss: 0.323240 | norm_perceptual_loss: 31.688204 [20:17:07.574015] [Epoch 2] | Iter: 50350 | Optim: 50350 | Time: 1.08s | LR: 0.000053 | SkipOpt: 0 loss: 0.207729 | l2_loss: 0.015226 | psnr: 18.174187 | perceptual_loss: 0.385006 | norm_perceptual_loss: 25.286383 [20:18:08.118771] [Epoch 2] | Iter: 50400 | Optim: 50400 | Time: 1.08s | LR: 0.000052 | SkipOpt: 0 loss: 0.153896 | l2_loss: 0.007056 | psnr: 21.514683 | perceptual_loss: 0.293681 | norm_perceptual_loss: 41.624004 [20:19:09.880925] [Epoch 2] | Iter: 50450 | Optim: 50450 | Time: 1.91s | LR: 0.000052 | SkipOpt: 0 loss: 0.215278 | l2_loss: 0.018701 | psnr: 17.281439 | perceptual_loss: 0.393155 | norm_perceptual_loss: 21.023609 [20:20:08.261108] [Epoch 2] | Iter: 50500 | Optim: 50500 | Time: 0.78s | LR: 0.000052 | SkipOpt: 0 loss: 0.202725 | l2_loss: 0.012564 | psnr: 19.008856 | perceptual_loss: 0.380324 | norm_perceptual_loss: 30.271841 [20:21:02.465171] [Epoch 2] | Iter: 50550 | Optim: 50550 | Time: 0.76s | LR: 0.000051 | SkipOpt: 0 loss: 0.190325 | l2_loss: 0.013281 | psnr: 18.767683 | perceptual_loss: 0.354087 | norm_perceptual_loss: 26.661112 [20:22:09.364185] [Epoch 2] | Iter: 50600 | Optim: 50600 | Time: 0.76s | LR: 0.000051 | SkipOpt: 0 loss: 0.172813 | l2_loss: 0.010956 | psnr: 19.603422 | perceptual_loss: 0.323714 | norm_perceptual_loss: 29.546307 [20:23:10.458183] [Epoch 2] | Iter: 50650 | Optim: 50650 | Time: 1.07s | LR: 0.000051 | SkipOpt: 0 loss: 0.176206 | l2_loss: 0.010703 | psnr: 19.704773 | perceptual_loss: 0.331004 | norm_perceptual_loss: 30.925076 [20:24:10.118244] [Epoch 2] | Iter: 50700 | Optim: 50700 | Time: 0.75s | LR: 0.000050 | SkipOpt: 0 loss: 0.187034 | l2_loss: 0.012198 | psnr: 19.137264 | perceptual_loss: 0.349672 | norm_perceptual_loss: 28.667355 [20:24:22.748514] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [20:25:15.835123] [Epoch 2] | Iter: 50750 | Optim: 50750 | Time: 0.75s | LR: 0.000050 | SkipOpt: 0 loss: 0.177524 | l2_loss: 0.010128 | psnr: 19.944641 | perceptual_loss: 0.334792 | norm_perceptual_loss: 33.055115 [20:26:21.462851] [Epoch 2] | Iter: 50800 | Optim: 50800 | Time: 1.91s | LR: 0.000050 | SkipOpt: 0 loss: 0.210547 | l2_loss: 0.014437 | psnr: 18.405354 | perceptual_loss: 0.392220 | norm_perceptual_loss: 27.168465 [20:27:24.661528] [Epoch 2] | Iter: 50850 | Optim: 50850 | Time: 1.91s | LR: 0.000049 | SkipOpt: 0 loss: 0.170391 | l2_loss: 0.012993 | psnr: 18.862841 | perceptual_loss: 0.314796 | norm_perceptual_loss: 24.227747 [20:28:28.959312] [Epoch 2] | Iter: 50900 | Optim: 50900 | Time: 1.07s | LR: 0.000049 | SkipOpt: 0 loss: 0.166660 | l2_loss: 0.011161 | psnr: 19.522831 | perceptual_loss: 0.310997 | norm_perceptual_loss: 27.863708 [20:29:30.109153] [Epoch 2] | Iter: 50950 | Optim: 50950 | Time: 0.78s | LR: 0.000049 | SkipOpt: 0 loss: 0.172137 | l2_loss: 0.009812 | psnr: 20.082214 | perceptual_loss: 0.324649 | norm_perceptual_loss: 33.085335 [20:30:33.700842] [Epoch 2] | Iter: 51000 | Optim: 51000 | Time: 1.09s | LR: 0.000048 | SkipOpt: 0 loss: 0.163550 | l2_loss: 0.008204 | psnr: 20.859737 | perceptual_loss: 0.310691 | norm_perceptual_loss: 37.870613 [20:30:41.216368] Saved checkpoint at iteration 51000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [20:30:41.216866] Saved latest checkpoint at iteration 51000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [20:31:38.072284] [Epoch 2] | Iter: 51050 | Optim: 51050 | Time: 0.78s | LR: 0.000048 | SkipOpt: 0 loss: 0.175390 | l2_loss: 0.009426 | psnr: 20.256565 | perceptual_loss: 0.331927 | norm_perceptual_loss: 35.212692 [20:32:38.200777] [Epoch 2] | Iter: 51100 | Optim: 51100 | Time: 1.91s | LR: 0.000048 | SkipOpt: 0 loss: 0.167248 | l2_loss: 0.009814 | psnr: 20.081472 | perceptual_loss: 0.314867 | norm_perceptual_loss: 32.082935 [20:33:39.761919] [Epoch 2] | Iter: 51150 | Optim: 51150 | Time: 1.08s | LR: 0.000047 | SkipOpt: 0 loss: 0.181988 | l2_loss: 0.011165 | psnr: 19.521345 | perceptual_loss: 0.341645 | norm_perceptual_loss: 30.599180 [20:34:39.892410] [Epoch 2] | Iter: 51200 | Optim: 51200 | Time: 1.91s | LR: 0.000047 | SkipOpt: 0 loss: 0.200432 | l2_loss: 0.013246 | psnr: 18.779007 | perceptual_loss: 0.374372 | norm_perceptual_loss: 28.262051 [20:35:42.009544] [Epoch 2] | Iter: 51250 | Optim: 51250 | Time: 1.91s | LR: 0.000047 | SkipOpt: 0 loss: 0.200646 | l2_loss: 0.013505 | psnr: 18.694975 | perceptual_loss: 0.374282 | norm_perceptual_loss: 27.713804 [20:36:50.724194] [Epoch 2] | Iter: 51300 | Optim: 51300 | Time: 1.91s | LR: 0.000046 | SkipOpt: 0 loss: 0.213613 | l2_loss: 0.018644 | psnr: 17.294725 | perceptual_loss: 0.389940 | norm_perceptual_loss: 20.915588 [20:37:47.322495] [Epoch 2] | Iter: 51350 | Optim: 51350 | Time: 1.07s | LR: 0.000046 | SkipOpt: 0 loss: 0.182285 | l2_loss: 0.012564 | psnr: 19.008648 | perceptual_loss: 0.339441 | norm_perceptual_loss: 27.016485 [20:38:49.862236] [Epoch 2] | Iter: 51400 | Optim: 51400 | Time: 1.08s | LR: 0.000046 | SkipOpt: 0 loss: 0.221560 | l2_loss: 0.015491 | psnr: 18.099167 | perceptual_loss: 0.412137 | norm_perceptual_loss: 26.604710 [20:39:53.531903] [Epoch 2] | Iter: 51450 | Optim: 51450 | Time: 1.91s | LR: 0.000045 | SkipOpt: 0 loss: 0.205186 | l2_loss: 0.016361 | psnr: 17.861824 | perceptual_loss: 0.377650 | norm_perceptual_loss: 23.081934 [20:40:52.454480] [Epoch 2] | Iter: 51500 | Optim: 51500 | Time: 1.91s | LR: 0.000045 | SkipOpt: 0 loss: 0.233837 | l2_loss: 0.019694 | psnr: 17.056734 | perceptual_loss: 0.428287 | norm_perceptual_loss: 21.747435 [20:41:17.501172] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [20:41:52.544437] [Epoch 2] | Iter: 51550 | Optim: 51550 | Time: 0.75s | LR: 0.000045 | SkipOpt: 0 loss: 0.182943 | l2_loss: 0.008770 | psnr: 20.570219 | perceptual_loss: 0.348347 | norm_perceptual_loss: 39.722313 [20:42:54.158950] [Epoch 2] | Iter: 51600 | Optim: 51600 | Time: 1.92s | LR: 0.000044 | SkipOpt: 0 loss: 0.194590 | l2_loss: 0.013990 | psnr: 18.541973 | perceptual_loss: 0.361201 | norm_perceptual_loss: 25.819393 [20:43:58.702700] [Epoch 2] | Iter: 51650 | Optim: 51650 | Time: 0.75s | LR: 0.000044 | SkipOpt: 0 loss: 0.150690 | l2_loss: 0.007646 | psnr: 21.165884 | perceptual_loss: 0.286089 | norm_perceptual_loss: 37.418755 [20:45:08.400031] [Epoch 2] | Iter: 51700 | Optim: 51700 | Time: 1.91s | LR: 0.000044 | SkipOpt: 0 loss: 0.181280 | l2_loss: 0.011711 | psnr: 19.314106 | perceptual_loss: 0.339138 | norm_perceptual_loss: 28.959251 [20:46:10.273216] [Epoch 2] | Iter: 51750 | Optim: 51750 | Time: 1.91s | LR: 0.000043 | SkipOpt: 0 loss: 0.203924 | l2_loss: 0.013325 | psnr: 18.753349 | perceptual_loss: 0.381198 | norm_perceptual_loss: 28.607824 [20:47:11.740084] [Epoch 2] | Iter: 51800 | Optim: 51800 | Time: 1.91s | LR: 0.000043 | SkipOpt: 0 loss: 0.204138 | l2_loss: 0.014058 | psnr: 18.520672 | perceptual_loss: 0.380159 | norm_perceptual_loss: 27.041578 [20:48:15.334166] [Epoch 2] | Iter: 51850 | Optim: 51850 | Time: 1.07s | LR: 0.000043 | SkipOpt: 0 loss: 0.180885 | l2_loss: 0.011558 | psnr: 19.371195 | perceptual_loss: 0.338655 | norm_perceptual_loss: 29.300638 [20:49:18.775583] [Epoch 2] | Iter: 51900 | Optim: 51900 | Time: 1.07s | LR: 0.000042 | SkipOpt: 0 loss: 0.215188 | l2_loss: 0.014338 | psnr: 18.435019 | perceptual_loss: 0.401699 | norm_perceptual_loss: 28.015774 [20:50:22.329484] [Epoch 2] | Iter: 51950 | Optim: 51950 | Time: 1.07s | LR: 0.000042 | SkipOpt: 0 loss: 0.149856 | l2_loss: 0.006475 | psnr: 21.887857 | perceptual_loss: 0.286763 | norm_perceptual_loss: 44.290329 [20:51:30.298826] [Epoch 2] | Iter: 52000 | Optim: 52000 | Time: 1.27s | LR: 0.000042 | SkipOpt: 0 loss: 0.181849 | l2_loss: 0.010679 | psnr: 19.714493 | perceptual_loss: 0.342340 | norm_perceptual_loss: 32.055775 [20:51:37.685097] Saved checkpoint at iteration 52000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [20:51:37.685585] Saved latest checkpoint at iteration 52000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [20:52:36.237158] [Epoch 2] | Iter: 52050 | Optim: 52050 | Time: 1.07s | LR: 0.000042 | SkipOpt: 0 loss: 0.172097 | l2_loss: 0.010590 | psnr: 19.751089 | perceptual_loss: 0.323015 | norm_perceptual_loss: 30.502241 [20:53:33.171584] [Epoch 2] | Iter: 52100 | Optim: 52100 | Time: 0.75s | LR: 0.000041 | SkipOpt: 0 loss: 0.184450 | l2_loss: 0.010727 | psnr: 19.695137 | perceptual_loss: 0.347445 | norm_perceptual_loss: 32.389153 [20:54:39.382823] [Epoch 2] | Iter: 52150 | Optim: 52150 | Time: 1.91s | LR: 0.000041 | SkipOpt: 0 loss: 0.183906 | l2_loss: 0.014285 | psnr: 18.451317 | perceptual_loss: 0.339243 | norm_perceptual_loss: 23.748842 [20:55:36.256845] [Epoch 2] | Iter: 52200 | Optim: 52200 | Time: 1.27s | LR: 0.000041 | SkipOpt: 0 loss: 0.165976 | l2_loss: 0.010207 | psnr: 19.910976 | perceptual_loss: 0.311538 | norm_perceptual_loss: 30.521698 [20:56:39.769116] [Epoch 2] | Iter: 52250 | Optim: 52250 | Time: 0.85s | LR: 0.000040 | SkipOpt: 0 loss: 0.158647 | l2_loss: 0.008620 | psnr: 20.644955 | perceptual_loss: 0.300054 | norm_perceptual_loss: 34.809265 [20:57:42.313207] [Epoch 2] | Iter: 52300 | Optim: 52300 | Time: 0.75s | LR: 0.000040 | SkipOpt: 0 loss: 0.160761 | l2_loss: 0.010090 | psnr: 19.960939 | perceptual_loss: 0.301342 | norm_perceptual_loss: 29.864374 [20:58:50.497840] [Epoch 2] | Iter: 52350 | Optim: 52350 | Time: 1.27s | LR: 0.000040 | SkipOpt: 0 loss: 0.154312 | l2_loss: 0.006879 | psnr: 21.625059 | perceptual_loss: 0.294868 | norm_perceptual_loss: 42.867939 [20:59:53.936806] [Epoch 2] | Iter: 52400 | Optim: 52400 | Time: 0.75s | LR: 0.000039 | SkipOpt: 0 loss: 0.165588 | l2_loss: 0.007119 | psnr: 21.475880 | perceptual_loss: 0.316939 | norm_perceptual_loss: 44.520889 [21:00:55.289385] [Epoch 2] | Iter: 52450 | Optim: 52450 | Time: 1.91s | LR: 0.000039 | SkipOpt: 0 loss: 0.191518 | l2_loss: 0.013595 | psnr: 18.666342 | perceptual_loss: 0.355847 | norm_perceptual_loss: 26.175631 [21:02:01.233680] [Epoch 2] | Iter: 52500 | Optim: 52500 | Time: 1.91s | LR: 0.000039 | SkipOpt: 0 loss: 0.190234 | l2_loss: 0.013271 | psnr: 18.771015 | perceptual_loss: 0.353926 | norm_perceptual_loss: 26.669436 [21:03:03.029872] [Epoch 2] | Iter: 52550 | Optim: 52550 | Time: 1.91s | LR: 0.000038 | SkipOpt: 0 loss: 0.180708 | l2_loss: 0.011641 | psnr: 19.339945 | perceptual_loss: 0.338133 | norm_perceptual_loss: 29.045704 [21:04:01.682630] [Epoch 2] | Iter: 52600 | Optim: 52600 | Time: 0.75s | LR: 0.000038 | SkipOpt: 0 loss: 0.170966 | l2_loss: 0.008495 | psnr: 20.708389 | perceptual_loss: 0.324943 | norm_perceptual_loss: 38.251301 [21:05:04.565716] [Epoch 2] | Iter: 52650 | Optim: 52650 | Time: 0.75s | LR: 0.000038 | SkipOpt: 0 loss: 0.172439 | l2_loss: 0.009656 | psnr: 20.151920 | perceptual_loss: 0.325566 | norm_perceptual_loss: 33.715603 [21:06:06.625449] [Epoch 2] | Iter: 52700 | Optim: 52700 | Time: 1.27s | LR: 0.000038 | SkipOpt: 0 loss: 0.172923 | l2_loss: 0.014127 | psnr: 18.499529 | perceptual_loss: 0.317591 | norm_perceptual_loss: 22.481295 [21:06:56.639463] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [21:07:11.899879] [Epoch 2] | Iter: 52750 | Optim: 52750 | Time: 1.91s | LR: 0.000037 | SkipOpt: 0 loss: 0.210549 | l2_loss: 0.017422 | psnr: 17.589037 | perceptual_loss: 0.386254 | norm_perceptual_loss: 22.170563 [21:08:12.034266] [Epoch 2] | Iter: 52800 | Optim: 52800 | Time: 0.75s | LR: 0.000037 | SkipOpt: 0 loss: 0.182215 | l2_loss: 0.013329 | psnr: 18.751957 | perceptual_loss: 0.337772 | norm_perceptual_loss: 25.340736 [21:09:10.163153] [Epoch 2] | Iter: 52850 | Optim: 52850 | Time: 1.91s | LR: 0.000037 | SkipOpt: 0 loss: 0.186150 | l2_loss: 0.011884 | psnr: 19.250286 | perceptual_loss: 0.348531 | norm_perceptual_loss: 29.327160 [21:10:10.580477] [Epoch 2] | Iter: 52900 | Optim: 52900 | Time: 1.08s | LR: 0.000036 | SkipOpt: 0 loss: 0.154744 | l2_loss: 0.007517 | psnr: 21.239304 | perceptual_loss: 0.294453 | norm_perceptual_loss: 39.169304 [21:11:10.929187] [Epoch 2] | Iter: 52950 | Optim: 52950 | Time: 1.07s | LR: 0.000036 | SkipOpt: 0 loss: 0.180718 | l2_loss: 0.010034 | psnr: 19.985054 | perceptual_loss: 0.341367 | norm_perceptual_loss: 34.019402 [21:12:11.090540] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [21:12:15.927533] [Epoch 2] | Iter: 53000 | Optim: 53000 | Time: 0.75s | LR: 0.000036 | SkipOpt: 0 loss: 0.175975 | l2_loss: 0.008158 | psnr: 20.884315 | perceptual_loss: 0.335634 | norm_perceptual_loss: 41.143173 [21:12:22.092967] Saved checkpoint at iteration 53000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [21:12:22.100545] Saved latest checkpoint at iteration 53000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [21:13:25.391344] [Epoch 2] | Iter: 53050 | Optim: 53050 | Time: 0.75s | LR: 0.000036 | SkipOpt: 0 loss: 0.155322 | l2_loss: 0.006380 | psnr: 21.952095 | perceptual_loss: 0.297886 | norm_perceptual_loss: 46.693790 [21:14:28.116102] [Epoch 2] | Iter: 53100 | Optim: 53100 | Time: 1.91s | LR: 0.000035 | SkipOpt: 0 loss: 0.182625 | l2_loss: 0.010573 | psnr: 19.758133 | perceptual_loss: 0.344105 | norm_perceptual_loss: 32.546501 [21:15:24.961949] [Epoch 2] | Iter: 53150 | Optim: 53150 | Time: 1.27s | LR: 0.000035 | SkipOpt: 0 loss: 0.176440 | l2_loss: 0.010241 | psnr: 19.896553 | perceptual_loss: 0.332399 | norm_perceptual_loss: 32.457455 [21:16:23.965133] [Epoch 2] | Iter: 53200 | Optim: 53200 | Time: 1.07s | LR: 0.000035 | SkipOpt: 0 loss: 0.153749 | l2_loss: 0.007127 | psnr: 21.471210 | perceptual_loss: 0.293245 | norm_perceptual_loss: 41.148262 [21:17:21.690988] [Epoch 2] | Iter: 53250 | Optim: 53250 | Time: 0.85s | LR: 0.000034 | SkipOpt: 0 loss: 0.206192 | l2_loss: 0.013936 | psnr: 18.558664 | perceptual_loss: 0.384512 | norm_perceptual_loss: 27.591583 [21:18:26.523940] [Epoch 2] | Iter: 53300 | Optim: 53300 | Time: 1.07s | LR: 0.000034 | SkipOpt: 0 loss: 0.197376 | l2_loss: 0.013276 | psnr: 18.769184 | perceptual_loss: 0.368199 | norm_perceptual_loss: 27.733274 [21:19:25.826053] [Epoch 2] | Iter: 53350 | Optim: 53350 | Time: 1.90s | LR: 0.000034 | SkipOpt: 0 loss: 0.184288 | l2_loss: 0.012241 | psnr: 19.121950 | perceptual_loss: 0.344095 | norm_perceptual_loss: 28.110836 [21:20:30.053390] [Epoch 2] | Iter: 53400 | Optim: 53400 | Time: 1.27s | LR: 0.000034 | SkipOpt: 0 loss: 0.203609 | l2_loss: 0.012883 | psnr: 18.899769 | perceptual_loss: 0.381453 | norm_perceptual_loss: 29.608553 [21:21:31.150425] [Epoch 2] | Iter: 53450 | Optim: 53450 | Time: 1.07s | LR: 0.000033 | SkipOpt: 0 loss: 0.161694 | l2_loss: 0.009667 | psnr: 20.147076 | perceptual_loss: 0.304055 | norm_perceptual_loss: 31.452806 [21:22:28.855449] [Epoch 2] | Iter: 53500 | Optim: 53500 | Time: 0.75s | LR: 0.000033 | SkipOpt: 0 loss: 0.146482 | l2_loss: 0.007566 | psnr: 21.211073 | perceptual_loss: 0.277832 | norm_perceptual_loss: 36.718834 [21:23:33.103172] [Epoch 2] | Iter: 53550 | Optim: 53550 | Time: 1.91s | LR: 0.000033 | SkipOpt: 0 loss: 0.202486 | l2_loss: 0.013231 | psnr: 18.784061 | perceptual_loss: 0.378510 | norm_perceptual_loss: 28.607759 [21:24:32.977540] [Epoch 2] | Iter: 53600 | Optim: 53600 | Time: 0.85s | LR: 0.000032 | SkipOpt: 0 loss: 0.229554 | l2_loss: 0.016131 | psnr: 17.923262 | perceptual_loss: 0.426845 | norm_perceptual_loss: 26.460375 [21:25:27.395278] [Epoch 2] | Iter: 53650 | Optim: 53650 | Time: 0.75s | LR: 0.000032 | SkipOpt: 0 loss: 0.154825 | l2_loss: 0.006678 | psnr: 21.753540 | perceptual_loss: 0.296294 | norm_perceptual_loss: 44.368725 [21:26:34.588793] [Epoch 2] | Iter: 53700 | Optim: 53700 | Time: 1.27s | LR: 0.000032 | SkipOpt: 0 loss: 0.191120 | l2_loss: 0.012059 | psnr: 19.186863 | perceptual_loss: 0.358122 | norm_perceptual_loss: 29.697315 [21:27:42.244558] [Epoch 2] | Iter: 53750 | Optim: 53750 | Time: 4.04s | LR: 0.000032 | SkipOpt: 0 loss: 0.189540 | l2_loss: 0.013582 | psnr: 18.670410 | perceptual_loss: 0.351917 | norm_perceptual_loss: 25.910851 [21:28:47.982099] [Epoch 2] | Iter: 53800 | Optim: 53800 | Time: 0.76s | LR: 0.000031 | SkipOpt: 0 loss: 0.202875 | l2_loss: 0.012280 | psnr: 19.107889 | perceptual_loss: 0.381189 | norm_perceptual_loss: 31.040552 [21:29:50.691900] [Epoch 2] | Iter: 53850 | Optim: 53850 | Time: 1.27s | LR: 0.000031 | SkipOpt: 0 loss: 0.148088 | l2_loss: 0.009588 | psnr: 20.182648 | perceptual_loss: 0.276999 | norm_perceptual_loss: 28.889690 [21:30:53.231453] [Epoch 2] | Iter: 53900 | Optim: 53900 | Time: 1.91s | LR: 0.000031 | SkipOpt: 0 loss: 0.183276 | l2_loss: 0.010809 | psnr: 19.662260 | perceptual_loss: 0.344936 | norm_perceptual_loss: 31.912714 [21:31:52.796402] [Epoch 2] | Iter: 53950 | Optim: 53950 | Time: 0.85s | LR: 0.000031 | SkipOpt: 0 loss: 0.174761 | l2_loss: 0.010329 | psnr: 19.859425 | perceptual_loss: 0.328864 | norm_perceptual_loss: 31.838982 [21:32:55.082887] [Epoch 2] | Iter: 54000 | Optim: 54000 | Time: 1.07s | LR: 0.000030 | SkipOpt: 0 loss: 0.163752 | l2_loss: 0.009857 | psnr: 20.062466 | perceptual_loss: 0.307790 | norm_perceptual_loss: 31.224907 [21:33:00.523470] Saved checkpoint at iteration 54000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [21:33:00.523844] Saved latest checkpoint at iteration 54000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [21:34:02.722887] [Epoch 2] | Iter: 54050 | Optim: 54050 | Time: 0.75s | LR: 0.000030 | SkipOpt: 0 loss: 0.165993 | l2_loss: 0.011294 | psnr: 19.471474 | perceptual_loss: 0.309398 | norm_perceptual_loss: 27.394590 [21:35:04.121087] [Epoch 2] | Iter: 54100 | Optim: 54100 | Time: 1.07s | LR: 0.000030 | SkipOpt: 0 loss: 0.132983 | l2_loss: 0.006453 | psnr: 21.902084 | perceptual_loss: 0.253060 | norm_perceptual_loss: 39.213120 [21:36:15.078439] [Epoch 2] | Iter: 54150 | Optim: 54150 | Time: 0.85s | LR: 0.000029 | SkipOpt: 0 loss: 0.169278 | l2_loss: 0.011230 | psnr: 19.496080 | perceptual_loss: 0.316095 | norm_perceptual_loss: 28.146559 [21:37:14.087659] [Epoch 2] | Iter: 54200 | Optim: 54200 | Time: 0.75s | LR: 0.000029 | SkipOpt: 0 loss: 0.184794 | l2_loss: 0.011001 | psnr: 19.585514 | perceptual_loss: 0.347586 | norm_perceptual_loss: 31.594660 [21:38:25.287104] [Epoch 2] | Iter: 54250 | Optim: 54250 | Time: 1.27s | LR: 0.000029 | SkipOpt: 0 loss: 0.206608 | l2_loss: 0.014403 | psnr: 18.415483 | perceptual_loss: 0.384409 | norm_perceptual_loss: 26.689615 [21:39:23.349098] [Epoch 2] | Iter: 54300 | Optim: 54300 | Time: 1.09s | LR: 0.000029 | SkipOpt: 0 loss: 0.188917 | l2_loss: 0.009824 | psnr: 20.077251 | perceptual_loss: 0.358187 | norm_perceptual_loss: 36.461544 [21:40:27.994319] [Epoch 2] | Iter: 54350 | Optim: 54350 | Time: 1.27s | LR: 0.000028 | SkipOpt: 0 loss: 0.180641 | l2_loss: 0.014616 | psnr: 18.351572 | perceptual_loss: 0.332048 | norm_perceptual_loss: 22.717394 [21:41:31.696971] [Epoch 2] | Iter: 54400 | Optim: 54400 | Time: 1.91s | LR: 0.000028 | SkipOpt: 0 loss: 0.183267 | l2_loss: 0.010769 | psnr: 19.678087 | perceptual_loss: 0.344995 | norm_perceptual_loss: 32.034775 [21:42:30.804158] [Epoch 2] | Iter: 54450 | Optim: 54450 | Time: 1.08s | LR: 0.000028 | SkipOpt: 0 loss: 0.170312 | l2_loss: 0.009816 | psnr: 20.080788 | perceptual_loss: 0.320993 | norm_perceptual_loss: 32.701962 [21:43:31.770011] [Epoch 2] | Iter: 54500 | Optim: 54500 | Time: 1.07s | LR: 0.000028 | SkipOpt: 0 loss: 0.161077 | l2_loss: 0.008286 | psnr: 20.816696 | perceptual_loss: 0.305582 | norm_perceptual_loss: 36.880581 [21:44:40.172972] [Epoch 2] | Iter: 54550 | Optim: 54550 | Time: 1.08s | LR: 0.000027 | SkipOpt: 0 loss: 0.173593 | l2_loss: 0.012571 | psnr: 19.006182 | perceptual_loss: 0.322044 | norm_perceptual_loss: 25.617315 [21:45:41.920604] [Epoch 2] | Iter: 54600 | Optim: 54600 | Time: 1.08s | LR: 0.000027 | SkipOpt: 0 loss: 0.152329 | l2_loss: 0.006610 | psnr: 21.797773 | perceptual_loss: 0.291436 | norm_perceptual_loss: 44.088074 [21:46:41.052591] [Epoch 2] | Iter: 54650 | Optim: 54650 | Time: 1.08s | LR: 0.000027 | SkipOpt: 0 loss: 0.172619 | l2_loss: 0.008777 | psnr: 20.566608 | perceptual_loss: 0.327685 | norm_perceptual_loss: 37.335133 [21:47:40.109200] [Epoch 2] | Iter: 54700 | Optim: 54700 | Time: 0.76s | LR: 0.000027 | SkipOpt: 0 loss: 0.163254 | l2_loss: 0.007040 | psnr: 21.524109 | perceptual_loss: 0.312428 | norm_perceptual_loss: 44.377266 [21:48:44.886486] [Epoch 2] | Iter: 54750 | Optim: 54750 | Time: 1.91s | LR: 0.000026 | SkipOpt: 0 loss: 0.201977 | l2_loss: 0.015151 | psnr: 18.195562 | perceptual_loss: 0.373652 | norm_perceptual_loss: 24.661730 [21:49:44.361714] [Epoch 2] | Iter: 54800 | Optim: 54800 | Time: 1.09s | LR: 0.000026 | SkipOpt: 0 loss: 0.158408 | l2_loss: 0.007770 | psnr: 21.095684 | perceptual_loss: 0.301275 | norm_perceptual_loss: 38.773235 [21:50:47.235412] [Epoch 2] | Iter: 54850 | Optim: 54850 | Time: 0.75s | LR: 0.000026 | SkipOpt: 0 loss: 0.165109 | l2_loss: 0.005567 | psnr: 22.544155 | perceptual_loss: 0.319085 | norm_perceptual_loss: 57.322128 [21:51:48.619583] [Epoch 2] | Iter: 54900 | Optim: 54900 | Time: 0.75s | LR: 0.000026 | SkipOpt: 0 loss: 0.180336 | l2_loss: 0.007761 | psnr: 21.100874 | perceptual_loss: 0.345150 | norm_perceptual_loss: 44.472874 [21:52:58.612748] [Epoch 2] | Iter: 54950 | Optim: 54950 | Time: 1.91s | LR: 0.000025 | SkipOpt: 0 loss: 0.213006 | l2_loss: 0.014157 | psnr: 18.490435 | perceptual_loss: 0.397698 | norm_perceptual_loss: 28.092934 [21:54:05.027721] [Epoch 2] | Iter: 55000 | Optim: 55000 | Time: 1.91s | LR: 0.000025 | SkipOpt: 0 loss: 0.178800 | l2_loss: 0.014543 | psnr: 18.373569 | perceptual_loss: 0.328514 | norm_perceptual_loss: 22.589716 [21:54:06.890601] Found 140 sequences [21:54:29.406867] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [21:54:30.699869] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [21:55:23.613927] Gathering scores from all processes... [21:55:23.614921] Padding local tensor of length 140 to max count: 140 [21:55:23.615500] Padding local tensor of length 140 to max count: 140 [21:55:23.616003] Padding local tensor of length 140 to max count: 140 [21:55:23.616308] Gathering scene names from all processes... [21:55:23.616455] Padding local tensor of length 140 on rank 0 to max count: 140 [21:55:23.617479] Evaluation metrics: [21:55:23.617678] psnr: 17.6170 [21:55:23.617777] ssim: 0.5025 [21:55:23.617867] lpips: 0.4760 [21:55:23.621809] Found 140 sequences [21:55:42.824207] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [21:55:43.696981] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [21:56:37.420625] Gathering scores from all processes... [21:56:37.422888] Padding local tensor of length 140 to max count: 140 [21:56:37.424241] Padding local tensor of length 140 to max count: 140 [21:56:37.425042] Padding local tensor of length 140 to max count: 140 [21:56:37.425558] Gathering scene names from all processes... [21:56:37.425906] Padding local tensor of length 140 on rank 0 to max count: 140 [21:56:37.427597] Evaluation metrics: [21:56:37.428050] psnr: 20.5774 [21:56:37.428291] ssim: 0.6019 [21:56:37.428504] lpips: 0.3623 [21:56:37.436113] Found 140 sequences [21:56:56.456696] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [21:56:57.915172] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [21:57:54.931858] Gathering scores from all processes... [21:57:54.934000] Padding local tensor of length 140 to max count: 140 [21:57:54.935163] Padding local tensor of length 140 to max count: 140 [21:57:54.935705] Padding local tensor of length 140 to max count: 140 [21:57:54.936292] Gathering scene names from all processes... [21:57:54.936539] Padding local tensor of length 140 on rank 0 to max count: 140 [21:57:54.937744] Evaluation metrics: [21:57:54.937959] psnr: 21.6602 [21:57:54.938056] ssim: 0.6385 [21:57:54.938229] lpips: 0.3245 [21:57:54.942064] [Eval][iter 55000] dl3dv_2view | psnr: 17.616961 | ssim: 0.502485 | lpips: 0.475959 || dl3dv_4view | psnr: 20.577383 | ssim: 0.601876 | lpips: 0.362314 || dl3dv_6view | psnr: 21.660151 | ssim: 0.638468 | lpips: 0.324519 [21:57:54.946252] [Eval][iter 55000] dl3dv_2view | psnr: 17.616961 | ssim: 0.502485 | lpips: 0.475959 || dl3dv_4view | psnr: 20.577383 | ssim: 0.601876 | lpips: 0.362314 || dl3dv_6view | psnr: 21.660151 | ssim: 0.638468 | lpips: 0.324519 [21:58:00.331910] Saved checkpoint at iteration 55000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [21:58:00.332298] Saved latest checkpoint at iteration 55000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [21:59:05.079953] [Epoch 2] | Iter: 55050 | Optim: 55050 | Time: 0.86s | LR: 0.000025 | SkipOpt: 0 loss: 0.165667 | l2_loss: 0.008918 | psnr: 20.497246 | perceptual_loss: 0.313498 | norm_perceptual_loss: 35.152767 [22:00:08.654464] [Epoch 2] | Iter: 55100 | Optim: 55100 | Time: 0.75s | LR: 0.000025 | SkipOpt: 0 loss: 0.148233 | l2_loss: 0.006830 | psnr: 21.655939 | perceptual_loss: 0.282807 | norm_perceptual_loss: 41.407967 [22:01:03.917361] [Epoch 2] | Iter: 55150 | Optim: 55150 | Time: 1.92s | LR: 0.000024 | SkipOpt: 0 loss: 0.203376 | l2_loss: 0.013612 | psnr: 18.660738 | perceptual_loss: 0.379528 | norm_perceptual_loss: 27.881575 [22:02:04.143028] [Epoch 2] | Iter: 55200 | Optim: 55200 | Time: 0.76s | LR: 0.000024 | SkipOpt: 0 loss: 0.154662 | l2_loss: 0.007212 | psnr: 21.419331 | perceptual_loss: 0.294900 | norm_perceptual_loss: 40.889179 [22:03:04.091146] [Epoch 2] | Iter: 55250 | Optim: 55250 | Time: 1.08s | LR: 0.000024 | SkipOpt: 0 loss: 0.180221 | l2_loss: 0.011324 | psnr: 19.460049 | perceptual_loss: 0.337795 | norm_perceptual_loss: 29.830328 [22:04:12.768261] [Epoch 2] | Iter: 55300 | Optim: 55300 | Time: 1.07s | LR: 0.000024 | SkipOpt: 0 loss: 0.158534 | l2_loss: 0.010010 | psnr: 19.995750 | perceptual_loss: 0.297048 | norm_perceptual_loss: 29.675753 [22:05:09.598722] [Epoch 2] | Iter: 55350 | Optim: 55350 | Time: 1.08s | LR: 0.000023 | SkipOpt: 0 loss: 0.183082 | l2_loss: 0.010030 | psnr: 19.987087 | perceptual_loss: 0.346104 | norm_perceptual_loss: 34.507675 [22:06:10.332365] [Epoch 2] | Iter: 55400 | Optim: 55400 | Time: 0.86s | LR: 0.000023 | SkipOpt: 0 loss: 0.145972 | l2_loss: 0.006109 | psnr: 22.140467 | perceptual_loss: 0.279726 | norm_perceptual_loss: 45.790874 [22:07:11.966774] [Epoch 2] | Iter: 55450 | Optim: 55450 | Time: 0.75s | LR: 0.000023 | SkipOpt: 0 loss: 0.167378 | l2_loss: 0.009121 | psnr: 20.399685 | perceptual_loss: 0.316514 | norm_perceptual_loss: 34.702507 [22:08:16.913145] [Epoch 2] | Iter: 55500 | Optim: 55500 | Time: 1.07s | LR: 0.000023 | SkipOpt: 0 loss: 0.184090 | l2_loss: 0.012597 | psnr: 18.997326 | perceptual_loss: 0.342986 | norm_perceptual_loss: 27.227551 [22:09:23.400143] [Epoch 2] | Iter: 55550 | Optim: 55550 | Time: 1.91s | LR: 0.000022 | SkipOpt: 0 loss: 0.212201 | l2_loss: 0.013735 | psnr: 18.621574 | perceptual_loss: 0.396931 | norm_perceptual_loss: 28.898338 [22:10:31.006270] [Epoch 2] | Iter: 55600 | Optim: 55600 | Time: 1.07s | LR: 0.000022 | SkipOpt: 0 loss: 0.179799 | l2_loss: 0.011193 | psnr: 19.510460 | perceptual_loss: 0.337212 | norm_perceptual_loss: 30.126505 [22:11:36.011529] [Epoch 2] | Iter: 55650 | Optim: 55650 | Time: 0.75s | LR: 0.000022 | SkipOpt: 0 loss: 0.216580 | l2_loss: 0.016765 | psnr: 17.755980 | perceptual_loss: 0.399629 | norm_perceptual_loss: 23.837198 [22:12:37.820816] [Epoch 2] | Iter: 55700 | Optim: 55700 | Time: 1.90s | LR: 0.000022 | SkipOpt: 0 loss: 0.203610 | l2_loss: 0.014107 | psnr: 18.505547 | perceptual_loss: 0.379005 | norm_perceptual_loss: 26.865778 [22:13:41.045548] [Epoch 2] | Iter: 55750 | Optim: 55750 | Time: 1.07s | LR: 0.000022 | SkipOpt: 0 loss: 0.145263 | l2_loss: 0.007645 | psnr: 21.166100 | perceptual_loss: 0.275236 | norm_perceptual_loss: 36.001064 [22:14:42.194402] [Epoch 2] | Iter: 55800 | Optim: 55800 | Time: 1.91s | LR: 0.000021 | SkipOpt: 0 loss: 0.208195 | l2_loss: 0.014686 | psnr: 18.330832 | perceptual_loss: 0.387018 | norm_perceptual_loss: 26.352037 [22:15:47.184935] [Epoch 2] | Iter: 55850 | Optim: 55850 | Time: 1.91s | LR: 0.000021 | SkipOpt: 0 loss: 0.195491 | l2_loss: 0.014211 | psnr: 18.473669 | perceptual_loss: 0.362560 | norm_perceptual_loss: 25.512123 [22:16:41.856648] [Epoch 2] | Iter: 55900 | Optim: 55900 | Time: 0.75s | LR: 0.000021 | SkipOpt: 0 loss: 0.171409 | l2_loss: 0.007957 | psnr: 20.992548 | perceptual_loss: 0.326904 | norm_perceptual_loss: 41.084167 [22:17:43.657184] [Epoch 2] | Iter: 55950 | Optim: 55950 | Time: 0.85s | LR: 0.000021 | SkipOpt: 0 loss: 0.164033 | l2_loss: 0.007479 | psnr: 21.261606 | perceptual_loss: 0.313108 | norm_perceptual_loss: 41.865311 [22:18:46.475256] [Epoch 2] | Iter: 56000 | Optim: 56000 | Time: 0.85s | LR: 0.000020 | SkipOpt: 0 loss: 0.137791 | l2_loss: 0.006054 | psnr: 22.179642 | perceptual_loss: 0.263475 | norm_perceptual_loss: 43.521477 [22:18:52.254956] Saved checkpoint at iteration 56000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [22:18:52.255395] Saved latest checkpoint at iteration 56000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [22:19:59.459560] [Epoch 2] | Iter: 56050 | Optim: 56050 | Time: 1.91s | LR: 0.000020 | SkipOpt: 0 loss: 0.190788 | l2_loss: 0.013016 | psnr: 18.855106 | perceptual_loss: 0.355543 | norm_perceptual_loss: 27.315102 [22:20:51.375175] [Epoch 2] | Iter: 56100 | Optim: 56100 | Time: 0.75s | LR: 0.000020 | SkipOpt: 0 loss: 0.148829 | l2_loss: 0.005483 | psnr: 22.609842 | perceptual_loss: 0.286692 | norm_perceptual_loss: 52.287777 [22:21:53.687197] [Epoch 2] | Iter: 56150 | Optim: 56150 | Time: 1.08s | LR: 0.000020 | SkipOpt: 0 loss: 0.136602 | l2_loss: 0.008505 | psnr: 20.703201 | perceptual_loss: 0.256194 | norm_perceptual_loss: 30.122314 [22:22:53.935666] [Epoch 2] | Iter: 56200 | Optim: 56200 | Time: 1.07s | LR: 0.000020 | SkipOpt: 0 loss: 0.183576 | l2_loss: 0.009818 | psnr: 20.079899 | perceptual_loss: 0.347517 | norm_perceptual_loss: 35.396950 [22:23:50.516318] [Epoch 2] | Iter: 56250 | Optim: 56250 | Time: 1.07s | LR: 0.000019 | SkipOpt: 0 loss: 0.184306 | l2_loss: 0.012082 | psnr: 19.178770 | perceptual_loss: 0.344448 | norm_perceptual_loss: 28.510239 [22:24:54.016810] [Epoch 2] | Iter: 56300 | Optim: 56300 | Time: 0.75s | LR: 0.000019 | SkipOpt: 0 loss: 0.192238 | l2_loss: 0.012957 | psnr: 18.874985 | perceptual_loss: 0.358562 | norm_perceptual_loss: 27.673399 [22:25:59.589985] [Epoch 2] | Iter: 56350 | Optim: 56350 | Time: 1.91s | LR: 0.000019 | SkipOpt: 0 loss: 0.194004 | l2_loss: 0.014869 | psnr: 18.277126 | perceptual_loss: 0.358269 | norm_perceptual_loss: 24.094711 [22:27:02.027026] [Epoch 2] | Iter: 56400 | Optim: 56400 | Time: 0.75s | LR: 0.000019 | SkipOpt: 0 loss: 0.232032 | l2_loss: 0.014067 | psnr: 18.517847 | perceptual_loss: 0.435929 | norm_perceptual_loss: 30.988518 [22:28:00.043189] [Epoch 2] | Iter: 56450 | Optim: 56450 | Time: 0.85s | LR: 0.000018 | SkipOpt: 0 loss: 0.161009 | l2_loss: 0.010614 | psnr: 19.741278 | perceptual_loss: 0.300791 | norm_perceptual_loss: 28.339535 [22:29:02.592995] [Epoch 2] | Iter: 56500 | Optim: 56500 | Time: 1.07s | LR: 0.000018 | SkipOpt: 0 loss: 0.166664 | l2_loss: 0.009738 | psnr: 20.115244 | perceptual_loss: 0.313853 | norm_perceptual_loss: 32.229240 [22:30:09.957334] [Epoch 2] | Iter: 56550 | Optim: 56550 | Time: 0.75s | LR: 0.000018 | SkipOpt: 0 loss: 0.182400 | l2_loss: 0.010908 | psnr: 19.622381 | perceptual_loss: 0.342984 | norm_perceptual_loss: 31.442127 [22:31:11.078954] [Epoch 2] | Iter: 56600 | Optim: 56600 | Time: 1.91s | LR: 0.000018 | SkipOpt: 0 loss: 0.221745 | l2_loss: 0.014534 | psnr: 18.376289 | perceptual_loss: 0.414422 | norm_perceptual_loss: 28.514906 [22:32:14.042077] [Epoch 2] | Iter: 56650 | Optim: 56650 | Time: 1.07s | LR: 0.000018 | SkipOpt: 0 loss: 0.197358 | l2_loss: 0.010984 | psnr: 19.592503 | perceptual_loss: 0.372748 | norm_perceptual_loss: 33.936348 [22:33:12.116403] [Epoch 2] | Iter: 56700 | Optim: 56700 | Time: 0.75s | LR: 0.000017 | SkipOpt: 0 loss: 0.162128 | l2_loss: 0.008866 | psnr: 20.522921 | perceptual_loss: 0.306525 | norm_perceptual_loss: 34.574661 [22:34:11.500863] [Epoch 2] | Iter: 56750 | Optim: 56750 | Time: 0.86s | LR: 0.000017 | SkipOpt: 0 loss: 0.164803 | l2_loss: 0.009012 | psnr: 20.451811 | perceptual_loss: 0.311581 | norm_perceptual_loss: 34.574226 [22:35:11.033762] [Epoch 2] | Iter: 56800 | Optim: 56800 | Time: 1.90s | LR: 0.000017 | SkipOpt: 0 loss: 0.189571 | l2_loss: 0.012724 | psnr: 18.953840 | perceptual_loss: 0.353694 | norm_perceptual_loss: 27.797901 [22:36:13.538965] [Epoch 2] | Iter: 56850 | Optim: 56850 | Time: 1.07s | LR: 0.000017 | SkipOpt: 0 loss: 0.186693 | l2_loss: 0.008661 | psnr: 20.624409 | perceptual_loss: 0.356065 | norm_perceptual_loss: 41.112137 [22:37:15.842781] [Epoch 2] | Iter: 56900 | Optim: 56900 | Time: 1.07s | LR: 0.000017 | SkipOpt: 0 loss: 0.162051 | l2_loss: 0.007871 | psnr: 21.039463 | perceptual_loss: 0.308358 | norm_perceptual_loss: 39.174374 [22:38:19.879436] [Epoch 2] | Iter: 56950 | Optim: 56950 | Time: 1.07s | LR: 0.000016 | SkipOpt: 0 loss: 0.158064 | l2_loss: 0.010073 | psnr: 19.968578 | perceptual_loss: 0.295983 | norm_perceptual_loss: 29.384876 [22:39:22.278278] [Epoch 2] | Iter: 57000 | Optim: 57000 | Time: 1.27s | LR: 0.000016 | SkipOpt: 0 loss: 0.226001 | l2_loss: 0.016174 | psnr: 17.911737 | perceptual_loss: 0.419653 | norm_perceptual_loss: 25.945641 [22:39:29.155053] Saved checkpoint at iteration 57000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [22:39:29.155508] Saved latest checkpoint at iteration 57000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [22:40:23.910098] [Epoch 2] | Iter: 57050 | Optim: 57050 | Time: 1.91s | LR: 0.000016 | SkipOpt: 0 loss: 0.183377 | l2_loss: 0.010621 | psnr: 19.738539 | perceptual_loss: 0.345513 | norm_perceptual_loss: 32.532578 [22:41:24.184228] [Epoch 2] | Iter: 57100 | Optim: 57100 | Time: 0.75s | LR: 0.000016 | SkipOpt: 0 loss: 0.159373 | l2_loss: 0.010878 | psnr: 19.634632 | perceptual_loss: 0.296990 | norm_perceptual_loss: 27.302683 [22:42:27.796916] [Epoch 2] | Iter: 57150 | Optim: 57150 | Time: 0.76s | LR: 0.000016 | SkipOpt: 0 loss: 0.181753 | l2_loss: 0.014921 | psnr: 18.262081 | perceptual_loss: 0.333665 | norm_perceptual_loss: 22.362419 [22:43:32.575923] [Epoch 2] | Iter: 57200 | Optim: 57200 | Time: 1.91s | LR: 0.000015 | SkipOpt: 0 loss: 0.202803 | l2_loss: 0.014709 | psnr: 18.324158 | perceptual_loss: 0.376188 | norm_perceptual_loss: 25.575315 [22:44:34.542259] [Epoch 2] | Iter: 57250 | Optim: 57250 | Time: 1.08s | LR: 0.000015 | SkipOpt: 0 loss: 0.192078 | l2_loss: 0.009497 | psnr: 20.224104 | perceptual_loss: 0.365161 | norm_perceptual_loss: 38.449898 [22:45:39.734938] [Epoch 2] | Iter: 57300 | Optim: 57300 | Time: 1.27s | LR: 0.000015 | SkipOpt: 0 loss: 0.203337 | l2_loss: 0.012878 | psnr: 18.901457 | perceptual_loss: 0.380918 | norm_perceptual_loss: 29.578575 [22:46:38.163700] [Epoch 2] | Iter: 57350 | Optim: 57350 | Time: 1.08s | LR: 0.000015 | SkipOpt: 0 loss: 0.171444 | l2_loss: 0.008796 | psnr: 20.557329 | perceptual_loss: 0.325297 | norm_perceptual_loss: 36.983894 [22:47:35.875421] [Epoch 2] | Iter: 57400 | Optim: 57400 | Time: 0.75s | LR: 0.000015 | SkipOpt: 0 loss: 0.189709 | l2_loss: 0.014725 | psnr: 18.319540 | perceptual_loss: 0.349968 | norm_perceptual_loss: 23.767426 [22:48:31.765960] [Epoch 2] | Iter: 57450 | Optim: 57450 | Time: 1.91s | LR: 0.000014 | SkipOpt: 0 loss: 0.184143 | l2_loss: 0.011827 | psnr: 19.271294 | perceptual_loss: 0.344631 | norm_perceptual_loss: 29.139627 [22:49:29.516419] [Epoch 2] | Iter: 57500 | Optim: 57500 | Time: 0.75s | LR: 0.000014 | SkipOpt: 0 loss: 0.118979 | l2_loss: 0.003191 | psnr: 24.961163 | perceptual_loss: 0.231577 | norm_perceptual_loss: 72.578964 [22:50:33.792128] [Epoch 2] | Iter: 57550 | Optim: 57550 | Time: 1.91s | LR: 0.000014 | SkipOpt: 0 loss: 0.206071 | l2_loss: 0.014175 | psnr: 18.484673 | perceptual_loss: 0.383792 | norm_perceptual_loss: 27.074659 [22:51:35.820130] [Epoch 2] | Iter: 57600 | Optim: 57600 | Time: 0.85s | LR: 0.000014 | SkipOpt: 0 loss: 0.164953 | l2_loss: 0.009915 | psnr: 20.036934 | perceptual_loss: 0.310076 | norm_perceptual_loss: 31.272402 [22:52:43.951541] [Epoch 2] | Iter: 57650 | Optim: 57650 | Time: 1.07s | LR: 0.000014 | SkipOpt: 0 loss: 0.178617 | l2_loss: 0.008793 | psnr: 20.558544 | perceptual_loss: 0.339648 | norm_perceptual_loss: 38.626354 [22:53:49.411358] [Epoch 2] | Iter: 57700 | Optim: 57700 | Time: 1.91s | LR: 0.000014 | SkipOpt: 0 loss: 0.168251 | l2_loss: 0.009364 | psnr: 20.285484 | perceptual_loss: 0.317775 | norm_perceptual_loss: 33.936626 [22:54:49.057380] [Epoch 2] | Iter: 57750 | Optim: 57750 | Time: 0.75s | LR: 0.000013 | SkipOpt: 0 loss: 0.105885 | l2_loss: 0.003934 | psnr: 24.051346 | perceptual_loss: 0.203901 | norm_perceptual_loss: 51.826832 [22:55:51.453629] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [22:55:55.801328] [Epoch 2] | Iter: 57800 | Optim: 57800 | Time: 0.75s | LR: 0.000013 | SkipOpt: 0 loss: 0.155696 | l2_loss: 0.009594 | psnr: 20.180071 | perceptual_loss: 0.292204 | norm_perceptual_loss: 30.457428 [22:56:59.978109] [Epoch 2] | Iter: 57850 | Optim: 57850 | Time: 1.07s | LR: 0.000013 | SkipOpt: 0 loss: 0.200760 | l2_loss: 0.011492 | psnr: 19.396061 | perceptual_loss: 0.378537 | norm_perceptual_loss: 32.939289 [22:57:59.720177] [Epoch 2] | Iter: 57900 | Optim: 57900 | Time: 0.75s | LR: 0.000013 | SkipOpt: 0 loss: 0.131507 | l2_loss: 0.005252 | psnr: 22.796379 | perceptual_loss: 0.252510 | norm_perceptual_loss: 48.074657 [22:59:02.004934] [Epoch 2] | Iter: 57950 | Optim: 57950 | Time: 1.91s | LR: 0.000013 | SkipOpt: 0 loss: 0.186355 | l2_loss: 0.013924 | psnr: 18.562471 | perceptual_loss: 0.344863 | norm_perceptual_loss: 24.768181 [23:00:14.984174] [Epoch 2] | Iter: 58000 | Optim: 58000 | Time: 1.27s | LR: 0.000012 | SkipOpt: 0 loss: 0.172082 | l2_loss: 0.010830 | psnr: 19.653568 | perceptual_loss: 0.322502 | norm_perceptual_loss: 29.777599 [23:00:21.169142] Saved checkpoint at iteration 58000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [23:00:21.169555] Saved latest checkpoint at iteration 58000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [23:01:25.113611] [Epoch 2] | Iter: 58050 | Optim: 58050 | Time: 1.08s | LR: 0.000012 | SkipOpt: 0 loss: 0.184709 | l2_loss: 0.012842 | psnr: 18.913591 | perceptual_loss: 0.343734 | norm_perceptual_loss: 26.765869 [23:02:24.433656] [Epoch 2] | Iter: 58100 | Optim: 58100 | Time: 0.75s | LR: 0.000012 | SkipOpt: 0 loss: 0.169881 | l2_loss: 0.012313 | psnr: 19.096529 | perceptual_loss: 0.315137 | norm_perceptual_loss: 25.594833 [23:03:31.518348] [Epoch 2] | Iter: 58150 | Optim: 58150 | Time: 1.91s | LR: 0.000012 | SkipOpt: 0 loss: 0.215236 | l2_loss: 0.017959 | psnr: 17.457289 | perceptual_loss: 0.394555 | norm_perceptual_loss: 21.970293 [23:04:33.574297] [Epoch 2] | Iter: 58200 | Optim: 58200 | Time: 1.08s | LR: 0.000012 | SkipOpt: 0 loss: 0.182720 | l2_loss: 0.010146 | psnr: 19.936905 | perceptual_loss: 0.345148 | norm_perceptual_loss: 34.016949 [23:05:32.236943] [Epoch 2] | Iter: 58250 | Optim: 58250 | Time: 0.75s | LR: 0.000012 | SkipOpt: 0 loss: 0.142931 | l2_loss: 0.005135 | psnr: 22.894207 | perceptual_loss: 0.275592 | norm_perceptual_loss: 53.664452 [23:06:37.284738] [Epoch 2] | Iter: 58300 | Optim: 58300 | Time: 1.07s | LR: 0.000011 | SkipOpt: 0 loss: 0.168907 | l2_loss: 0.008609 | psnr: 20.650652 | perceptual_loss: 0.320597 | norm_perceptual_loss: 37.241280 [23:07:43.069271] [Epoch 2] | Iter: 58350 | Optim: 58350 | Time: 1.27s | LR: 0.000011 | SkipOpt: 0 loss: 0.195535 | l2_loss: 0.013633 | psnr: 18.654196 | perceptual_loss: 0.363805 | norm_perceptual_loss: 26.686314 [23:08:45.001123] [Epoch 2] | Iter: 58400 | Optim: 58400 | Time: 1.07s | LR: 0.000011 | SkipOpt: 0 loss: 0.153887 | l2_loss: 0.009941 | psnr: 20.025728 | perceptual_loss: 0.287893 | norm_perceptual_loss: 28.960318 [23:09:47.343445] [Epoch 2] | Iter: 58450 | Optim: 58450 | Time: 1.07s | LR: 0.000011 | SkipOpt: 0 loss: 0.166439 | l2_loss: 0.011274 | psnr: 19.479031 | perceptual_loss: 0.310330 | norm_perceptual_loss: 27.524952 [23:10:49.450866] [Epoch 2] | Iter: 58500 | Optim: 58500 | Time: 1.91s | LR: 0.000011 | SkipOpt: 0 loss: 0.205835 | l2_loss: 0.015499 | psnr: 18.096920 | perceptual_loss: 0.380671 | norm_perceptual_loss: 24.560768 [23:11:52.795410] [Epoch 2] | Iter: 58550 | Optim: 58550 | Time: 0.85s | LR: 0.000011 | SkipOpt: 0 loss: 0.158652 | l2_loss: 0.010997 | psnr: 19.587271 | perceptual_loss: 0.295310 | norm_perceptual_loss: 26.853806 [23:12:57.451618] [Epoch 2] | Iter: 58600 | Optim: 58600 | Time: 0.85s | LR: 0.000010 | SkipOpt: 0 loss: 0.134272 | l2_loss: 0.005517 | psnr: 22.582895 | perceptual_loss: 0.257509 | norm_perceptual_loss: 46.674793 [23:13:56.964121] [Epoch 2] | Iter: 58650 | Optim: 58650 | Time: 1.07s | LR: 0.000010 | SkipOpt: 0 loss: 0.214912 | l2_loss: 0.015410 | psnr: 18.122110 | perceptual_loss: 0.399006 | norm_perceptual_loss: 25.893465 [23:15:00.936744] [Epoch 2] | Iter: 58700 | Optim: 58700 | Time: 1.91s | LR: 0.000010 | SkipOpt: 0 loss: 0.174332 | l2_loss: 0.010269 | psnr: 19.884642 | perceptual_loss: 0.328125 | norm_perceptual_loss: 31.952456 [23:16:01.404979] [Epoch 2] | Iter: 58750 | Optim: 58750 | Time: 1.91s | LR: 0.000010 | SkipOpt: 0 loss: 0.191905 | l2_loss: 0.013229 | psnr: 18.784645 | perceptual_loss: 0.357351 | norm_perceptual_loss: 27.012136 [23:16:58.864120] [Epoch 2] | Iter: 58800 | Optim: 58800 | Time: 1.07s | LR: 0.000010 | SkipOpt: 0 loss: 0.164909 | l2_loss: 0.010113 | psnr: 19.951244 | perceptual_loss: 0.309593 | norm_perceptual_loss: 30.613649 [23:17:53.505109] [Epoch 2] | Iter: 58850 | Optim: 58850 | Time: 1.08s | LR: 0.000010 | SkipOpt: 0 loss: 0.147465 | l2_loss: 0.005635 | psnr: 22.491173 | perceptual_loss: 0.283660 | norm_perceptual_loss: 50.340305 [23:18:53.682849] [Epoch 2] | Iter: 58900 | Optim: 58900 | Time: 1.91s | LR: 0.000009 | SkipOpt: 0 loss: 0.183269 | l2_loss: 0.011371 | psnr: 19.441868 | perceptual_loss: 0.343796 | norm_perceptual_loss: 30.233450 [23:19:51.974226] [Epoch 2] | Iter: 58950 | Optim: 58950 | Time: 1.07s | LR: 0.000009 | SkipOpt: 0 loss: 0.198367 | l2_loss: 0.013775 | psnr: 18.609062 | perceptual_loss: 0.369183 | norm_perceptual_loss: 26.800833 [23:20:56.516631] [Epoch 2] | Iter: 59000 | Optim: 59000 | Time: 0.76s | LR: 0.000009 | SkipOpt: 0 loss: 0.201600 | l2_loss: 0.013714 | psnr: 18.628294 | perceptual_loss: 0.375772 | norm_perceptual_loss: 27.400194 [23:21:03.186637] Saved checkpoint at iteration 59000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [23:21:03.187039] Saved latest checkpoint at iteration 59000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [23:22:10.039617] [Epoch 2] | Iter: 59050 | Optim: 59050 | Time: 1.91s | LR: 0.000009 | SkipOpt: 0 loss: 0.192976 | l2_loss: 0.012358 | psnr: 19.080599 | perceptual_loss: 0.361237 | norm_perceptual_loss: 29.231569 [23:23:12.411230] [Epoch 2] | Iter: 59100 | Optim: 59100 | Time: 1.07s | LR: 0.000009 | SkipOpt: 0 loss: 0.181724 | l2_loss: 0.011125 | psnr: 19.536934 | perceptual_loss: 0.341198 | norm_perceptual_loss: 30.669046 [23:24:13.371960] [Epoch 2] | Iter: 59150 | Optim: 59150 | Time: 1.07s | LR: 0.000009 | SkipOpt: 0 loss: 0.170685 | l2_loss: 0.010991 | psnr: 19.589575 | perceptual_loss: 0.319388 | norm_perceptual_loss: 29.058708 [23:25:12.975989] [Epoch 2] | Iter: 59200 | Optim: 59200 | Time: 0.75s | LR: 0.000009 | SkipOpt: 0 loss: 0.152338 | l2_loss: 0.007290 | psnr: 21.372845 | perceptual_loss: 0.290097 | norm_perceptual_loss: 39.794979 [23:26:11.760479] [Epoch 2] | Iter: 59250 | Optim: 59250 | Time: 0.75s | LR: 0.000008 | SkipOpt: 0 loss: 0.123041 | l2_loss: 0.005270 | psnr: 22.781986 | perceptual_loss: 0.235541 | norm_perceptual_loss: 44.695698 [23:27:07.808071] [Epoch 2] | Iter: 59300 | Optim: 59300 | Time: 1.08s | LR: 0.000008 | SkipOpt: 0 loss: 0.191398 | l2_loss: 0.012578 | psnr: 19.003866 | perceptual_loss: 0.357640 | norm_perceptual_loss: 28.433611 [23:28:11.154289] [Epoch 2] | Iter: 59350 | Optim: 59350 | Time: 0.76s | LR: 0.000008 | SkipOpt: 0 loss: 0.170770 | l2_loss: 0.011132 | psnr: 19.534220 | perceptual_loss: 0.319276 | norm_perceptual_loss: 28.680588 [23:29:16.271679] [Epoch 2] | Iter: 59400 | Optim: 59400 | Time: 1.08s | LR: 0.000008 | SkipOpt: 0 loss: 0.204151 | l2_loss: 0.014656 | psnr: 18.339912 | perceptual_loss: 0.378991 | norm_perceptual_loss: 25.859484 [23:30:16.365245] [Epoch 2] | Iter: 59450 | Optim: 59450 | Time: 0.85s | LR: 0.000008 | SkipOpt: 0 loss: 0.176052 | l2_loss: 0.010386 | psnr: 19.835331 | perceptual_loss: 0.331331 | norm_perceptual_loss: 31.900370 [23:31:16.998688] [Epoch 2] | Iter: 59500 | Optim: 59500 | Time: 0.76s | LR: 0.000008 | SkipOpt: 0 loss: 0.155905 | l2_loss: 0.008891 | psnr: 20.510626 | perceptual_loss: 0.294028 | norm_perceptual_loss: 33.071247 [23:32:17.414707] [Epoch 2] | Iter: 59550 | Optim: 59550 | Time: 0.76s | LR: 0.000008 | SkipOpt: 0 loss: 0.185519 | l2_loss: 0.009985 | psnr: 20.006563 | perceptual_loss: 0.351068 | norm_perceptual_loss: 35.159863 [23:33:17.123615] [Epoch 2] | Iter: 59600 | Optim: 59600 | Time: 0.85s | LR: 0.000007 | SkipOpt: 0 loss: 0.150036 | l2_loss: 0.006659 | psnr: 21.765921 | perceptual_loss: 0.286754 | norm_perceptual_loss: 43.062752 [23:34:23.998015] [Epoch 2] | Iter: 59650 | Optim: 59650 | Time: 1.91s | LR: 0.000007 | SkipOpt: 0 loss: 0.160097 | l2_loss: 0.010064 | psnr: 19.972471 | perceptual_loss: 0.300067 | norm_perceptual_loss: 29.817129 [23:35:25.972269] [Epoch 2] | Iter: 59700 | Optim: 59700 | Time: 1.27s | LR: 0.000007 | SkipOpt: 0 loss: 0.171248 | l2_loss: 0.010169 | psnr: 19.927143 | perceptual_loss: 0.322157 | norm_perceptual_loss: 31.679796 [23:36:31.782844] [Epoch 2] | Iter: 59750 | Optim: 59750 | Time: 1.91s | LR: 0.000007 | SkipOpt: 0 loss: 0.213298 | l2_loss: 0.016096 | psnr: 17.932701 | perceptual_loss: 0.394403 | norm_perceptual_loss: 24.502472 [23:37:36.138837] [Epoch 2] | Iter: 59800 | Optim: 59800 | Time: 1.90s | LR: 0.000007 | SkipOpt: 0 loss: 0.175785 | l2_loss: 0.010905 | psnr: 19.623795 | perceptual_loss: 0.329760 | norm_perceptual_loss: 30.239664 [23:38:34.120522] [Epoch 2] | Iter: 59850 | Optim: 59850 | Time: 0.76s | LR: 0.000007 | SkipOpt: 0 loss: 0.200228 | l2_loss: 0.009704 | psnr: 20.130585 | perceptual_loss: 0.381048 | norm_perceptual_loss: 39.267971 [23:39:35.654537] [Epoch 2] | Iter: 59900 | Optim: 59900 | Time: 0.75s | LR: 0.000007 | SkipOpt: 0 loss: 0.170173 | l2_loss: 0.007915 | psnr: 21.015318 | perceptual_loss: 0.324516 | norm_perceptual_loss: 40.998440 [23:40:36.418860] [Epoch 2] | Iter: 59950 | Optim: 59950 | Time: 1.07s | LR: 0.000007 | SkipOpt: 0 loss: 0.134786 | l2_loss: 0.007185 | psnr: 21.435608 | perceptual_loss: 0.255203 | norm_perceptual_loss: 35.517788 [23:41:36.549990] [Epoch 2] | Iter: 60000 | Optim: 60000 | Time: 0.75s | LR: 0.000006 | SkipOpt: 0 loss: 0.162196 | l2_loss: 0.009820 | psnr: 20.078821 | perceptual_loss: 0.304751 | norm_perceptual_loss: 31.033211 [23:41:38.276434] Found 140 sequences [23:41:53.384418] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [23:41:54.676540] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [23:42:47.549651] Gathering scores from all processes... [23:42:47.550790] Padding local tensor of length 140 to max count: 140 [23:42:47.551437] Padding local tensor of length 140 to max count: 140 [23:42:47.551973] Padding local tensor of length 140 to max count: 140 [23:42:47.552325] Gathering scene names from all processes... [23:42:47.552454] Padding local tensor of length 140 on rank 0 to max count: 140 [23:42:47.553542] Evaluation metrics: [23:42:47.553740] psnr: 17.6271 [23:42:47.553828] ssim: 0.5029 [23:42:47.553927] lpips: 0.4757 [23:42:47.557907] Found 140 sequences [23:43:08.964922] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [23:43:10.149235] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [23:44:03.046748] Gathering scores from all processes... [23:44:03.048250] Padding local tensor of length 140 to max count: 140 [23:44:03.048851] Padding local tensor of length 140 to max count: 140 [23:44:03.049427] Padding local tensor of length 140 to max count: 140 [23:44:03.049761] Gathering scene names from all processes... [23:44:03.049924] Padding local tensor of length 140 on rank 0 to max count: 140 [23:44:03.051002] Evaluation metrics: [23:44:03.051201] psnr: 20.6623 [23:44:03.051296] ssim: 0.6058 [23:44:03.051391] lpips: 0.3587 [23:44:03.055304] Found 140 sequences [23:44:26.623627] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [23:44:27.764714] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [23:45:23.495925] Gathering scores from all processes... [23:45:23.496799] Padding local tensor of length 140 to max count: 140 [23:45:23.497439] Padding local tensor of length 140 to max count: 140 [23:45:23.497940] Padding local tensor of length 140 to max count: 140 [23:45:23.498253] Gathering scene names from all processes... [23:45:23.498398] Padding local tensor of length 140 on rank 0 to max count: 140 [23:45:23.499418] Evaluation metrics: [23:45:23.499629] psnr: 21.7510 [23:45:23.499736] ssim: 0.6429 [23:45:23.499829] lpips: 0.3206 [23:45:23.501501] [Eval][iter 60000] dl3dv_2view | psnr: 17.627119 | ssim: 0.502907 | lpips: 0.475652 || dl3dv_4view | psnr: 20.662294 | ssim: 0.605849 | lpips: 0.358700 || dl3dv_6view | psnr: 21.750998 | ssim: 0.642899 | lpips: 0.320617 [23:45:23.504509] [Eval][iter 60000] dl3dv_2view | psnr: 17.627119 | ssim: 0.502907 | lpips: 0.475652 || dl3dv_4view | psnr: 20.662294 | ssim: 0.605849 | lpips: 0.358700 || dl3dv_6view | psnr: 21.750998 | ssim: 0.642899 | lpips: 0.320617 [23:45:27.437491] Saved checkpoint at iteration 60000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [23:45:27.437940] Saved latest checkpoint at iteration 60000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [23:46:30.414612] [Epoch 2] | Iter: 60050 | Optim: 60050 | Time: 0.75s | LR: 0.000006 | SkipOpt: 0 loss: 0.177856 | l2_loss: 0.007122 | psnr: 21.474140 | perceptual_loss: 0.341468 | norm_perceptual_loss: 47.947216 [23:47:33.481085] [Epoch 2] | Iter: 60100 | Optim: 60100 | Time: 1.08s | LR: 0.000006 | SkipOpt: 0 loss: 0.155458 | l2_loss: 0.009562 | psnr: 20.194477 | perceptual_loss: 0.291791 | norm_perceptual_loss: 30.515471 [23:48:38.690833] [Epoch 2] | Iter: 60150 | Optim: 60150 | Time: 1.91s | LR: 0.000006 | SkipOpt: 0 loss: 0.199562 | l2_loss: 0.015074 | psnr: 18.217831 | perceptual_loss: 0.368977 | norm_perceptual_loss: 24.478359 [23:49:39.649200] [Epoch 2] | Iter: 60200 | Optim: 60200 | Time: 0.75s | LR: 0.000006 | SkipOpt: 0 loss: 0.166850 | l2_loss: 0.009037 | psnr: 20.439938 | perceptual_loss: 0.315626 | norm_perceptual_loss: 34.927418 [23:50:42.194244] [Epoch 2] | Iter: 60250 | Optim: 60250 | Time: 2.63s | LR: 0.000006 | SkipOpt: 0 loss: 0.156031 | l2_loss: 0.008460 | psnr: 20.726376 | perceptual_loss: 0.295143 | norm_perceptual_loss: 34.887524 [23:51:43.380762] [Epoch 2] | Iter: 60300 | Optim: 60300 | Time: 1.27s | LR: 0.000006 | SkipOpt: 0 loss: 0.185590 | l2_loss: 0.013258 | psnr: 18.775263 | perceptual_loss: 0.344664 | norm_perceptual_loss: 25.996897 [23:52:47.601229] [Epoch 2] | Iter: 60350 | Optim: 60350 | Time: 1.08s | LR: 0.000006 | SkipOpt: 0 loss: 0.175673 | l2_loss: 0.010742 | psnr: 19.689123 | perceptual_loss: 0.329861 | norm_perceptual_loss: 30.707397 [23:53:44.871366] [Epoch 2] | Iter: 60400 | Optim: 60400 | Time: 0.86s | LR: 0.000005 | SkipOpt: 0 loss: 0.148305 | l2_loss: 0.006697 | psnr: 21.740961 | perceptual_loss: 0.283215 | norm_perceptual_loss: 42.287510 [23:54:49.650773] [Epoch 2] | Iter: 60450 | Optim: 60450 | Time: 1.07s | LR: 0.000005 | SkipOpt: 0 loss: 0.187995 | l2_loss: 0.009485 | psnr: 20.229519 | perceptual_loss: 0.357019 | norm_perceptual_loss: 37.639412 [23:55:56.114230] [Epoch 2] | Iter: 60500 | Optim: 60500 | Time: 1.91s | LR: 0.000005 | SkipOpt: 0 loss: 0.198689 | l2_loss: 0.012726 | psnr: 18.953152 | perceptual_loss: 0.371927 | norm_perceptual_loss: 29.226240 [23:56:53.599640] [Epoch 2] | Iter: 60550 | Optim: 60550 | Time: 0.75s | LR: 0.000005 | SkipOpt: 0 loss: 0.141096 | l2_loss: 0.008152 | psnr: 20.887489 | perceptual_loss: 0.265888 | norm_perceptual_loss: 32.617317 [23:57:58.171251] [Epoch 2] | Iter: 60600 | Optim: 60600 | Time: 1.91s | LR: 0.000005 | SkipOpt: 0 loss: 0.202153 | l2_loss: 0.017832 | psnr: 17.488035 | perceptual_loss: 0.368643 | norm_perceptual_loss: 20.673283 [23:59:01.774261] [Epoch 2] | Iter: 60650 | Optim: 60650 | Time: 1.27s | LR: 0.000005 | SkipOpt: 0 loss: 0.189424 | l2_loss: 0.012721 | psnr: 18.954655 | perceptual_loss: 0.353405 | norm_perceptual_loss: 27.780399 [00:00:02.367832] [Epoch 2] | Iter: 60700 | Optim: 60700 | Time: 0.85s | LR: 0.000005 | SkipOpt: 0 loss: 0.197104 | l2_loss: 0.016295 | psnr: 17.879555 | perceptual_loss: 0.361619 | norm_perceptual_loss: 22.192516 [00:01:06.559220] [Epoch 2] | Iter: 60750 | Optim: 60750 | Time: 1.07s | LR: 0.000005 | SkipOpt: 0 loss: 0.178687 | l2_loss: 0.010623 | psnr: 19.737455 | perceptual_loss: 0.336128 | norm_perceptual_loss: 31.641043 [00:02:02.354309] [Epoch 2] | Iter: 60800 | Optim: 60800 | Time: 1.08s | LR: 0.000005 | SkipOpt: 0 loss: 0.241814 | l2_loss: 0.018791 | psnr: 17.260445 | perceptual_loss: 0.446045 | norm_perceptual_loss: 23.736864 [00:03:06.768157] [Epoch 2] | Iter: 60850 | Optim: 60850 | Time: 1.91s | LR: 0.000004 | SkipOpt: 0 loss: 0.179076 | l2_loss: 0.010981 | psnr: 19.593437 | perceptual_loss: 0.336189 | norm_perceptual_loss: 30.614532 [00:04:11.442256] [Epoch 2] | Iter: 60900 | Optim: 60900 | Time: 1.07s | LR: 0.000004 | SkipOpt: 0 loss: 0.158721 | l2_loss: 0.008142 | psnr: 20.892712 | perceptual_loss: 0.301157 | norm_perceptual_loss: 36.988316 [00:05:14.147759] [Epoch 2] | Iter: 60950 | Optim: 60950 | Time: 0.75s | LR: 0.000004 | SkipOpt: 0 loss: 0.149756 | l2_loss: 0.007296 | psnr: 21.368933 | perceptual_loss: 0.284919 | norm_perceptual_loss: 39.049442 [00:06:16.701107] [Epoch 2] | Iter: 61000 | Optim: 61000 | Time: 1.07s | LR: 0.000004 | SkipOpt: 0 loss: 0.154548 | l2_loss: 0.009391 | psnr: 20.272991 | perceptual_loss: 0.290314 | norm_perceptual_loss: 30.914864 [00:06:24.801051] Saved checkpoint at iteration 61000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [00:06:24.801445] Saved latest checkpoint at iteration 61000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [00:07:23.778575] [Epoch 2] | Iter: 61050 | Optim: 61050 | Time: 1.07s | LR: 0.000004 | SkipOpt: 0 loss: 0.247204 | l2_loss: 0.019695 | psnr: 17.056429 | perceptual_loss: 0.455018 | norm_perceptual_loss: 23.103151 [00:08:24.697635] [Epoch 2] | Iter: 61100 | Optim: 61100 | Time: 0.75s | LR: 0.000004 | SkipOpt: 0 loss: 0.112295 | l2_loss: 0.003318 | psnr: 24.791719 | perceptual_loss: 0.217955 | norm_perceptual_loss: 65.695877 [00:09:26.220924] [Epoch 2] | Iter: 61150 | Optim: 61150 | Time: 0.85s | LR: 0.000004 | SkipOpt: 0 loss: 0.132252 | l2_loss: 0.005936 | psnr: 22.265036 | perceptual_loss: 0.252631 | norm_perceptual_loss: 42.558922 [00:10:24.961759] [Epoch 2] | Iter: 61200 | Optim: 61200 | Time: 0.75s | LR: 0.000004 | SkipOpt: 0 loss: 0.189652 | l2_loss: 0.012487 | psnr: 19.035370 | perceptual_loss: 0.354330 | norm_perceptual_loss: 28.375603 [00:11:21.630827] [Epoch 2] | Iter: 61250 | Optim: 61250 | Time: 0.85s | LR: 0.000004 | SkipOpt: 0 loss: 0.129021 | l2_loss: 0.005561 | psnr: 22.548208 | perceptual_loss: 0.246918 | norm_perceptual_loss: 44.399120 [00:12:26.703189] [Epoch 2] | Iter: 61300 | Optim: 61300 | Time: 1.91s | LR: 0.000004 | SkipOpt: 0 loss: 0.216148 | l2_loss: 0.017591 | psnr: 17.547096 | perceptual_loss: 0.397113 | norm_perceptual_loss: 22.574816 [00:13:32.840842] [Epoch 2] | Iter: 61350 | Optim: 61350 | Time: 1.08s | LR: 0.000003 | SkipOpt: 0 loss: 0.192240 | l2_loss: 0.011716 | psnr: 19.312288 | perceptual_loss: 0.361048 | norm_perceptual_loss: 30.817196 [00:14:40.579693] [Epoch 2] | Iter: 61400 | Optim: 61400 | Time: 1.08s | LR: 0.000003 | SkipOpt: 0 loss: 0.183504 | l2_loss: 0.010729 | psnr: 19.694426 | perceptual_loss: 0.345550 | norm_perceptual_loss: 32.207230 [00:15:36.805277] [Epoch 2] | Iter: 61450 | Optim: 61450 | Time: 1.07s | LR: 0.000003 | SkipOpt: 0 loss: 0.137989 | l2_loss: 0.004930 | psnr: 23.071873 | perceptual_loss: 0.266119 | norm_perceptual_loss: 53.983719 [00:16:43.252698] [Epoch 2] | Iter: 61500 | Optim: 61500 | Time: 1.91s | LR: 0.000003 | SkipOpt: 0 loss: 0.196991 | l2_loss: 0.013066 | psnr: 18.838715 | perceptual_loss: 0.367850 | norm_perceptual_loss: 28.154137 [00:17:48.894026] [Epoch 2] | Iter: 61550 | Optim: 61550 | Time: 0.75s | LR: 0.000003 | SkipOpt: 0 loss: 0.140324 | l2_loss: 0.006503 | psnr: 21.868816 | perceptual_loss: 0.267642 | norm_perceptual_loss: 41.156178 [00:18:48.424079] [Epoch 2] | Iter: 61600 | Optim: 61600 | Time: 0.75s | LR: 0.000003 | SkipOpt: 0 loss: 0.167342 | l2_loss: 0.006037 | psnr: 22.191833 | perceptual_loss: 0.322610 | norm_perceptual_loss: 53.439423 [00:19:49.463772] [Epoch 2] | Iter: 61650 | Optim: 61650 | Time: 1.08s | LR: 0.000003 | SkipOpt: 0 loss: 0.159281 | l2_loss: 0.007938 | psnr: 21.002708 | perceptual_loss: 0.302685 | norm_perceptual_loss: 38.129520 [00:20:57.705719] [Epoch 2] | Iter: 61700 | Optim: 61700 | Time: 1.08s | LR: 0.000003 | SkipOpt: 0 loss: 0.184593 | l2_loss: 0.012071 | psnr: 19.182514 | perceptual_loss: 0.345044 | norm_perceptual_loss: 28.584188 [00:21:56.826297] [Epoch 2] | Iter: 61750 | Optim: 61750 | Time: 1.07s | LR: 0.000003 | SkipOpt: 0 loss: 0.180138 | l2_loss: 0.010873 | psnr: 19.636663 | perceptual_loss: 0.338530 | norm_perceptual_loss: 31.136055 [00:22:56.896570] [Epoch 2] | Iter: 61800 | Optim: 61800 | Time: 0.75s | LR: 0.000003 | SkipOpt: 0 loss: 0.182318 | l2_loss: 0.011380 | psnr: 19.438454 | perceptual_loss: 0.341876 | norm_perceptual_loss: 30.040955 [00:23:53.878247] [Epoch 2] | Iter: 61850 | Optim: 61850 | Time: 1.92s | LR: 0.000003 | SkipOpt: 0 loss: 0.190152 | l2_loss: 0.011993 | psnr: 19.210793 | perceptual_loss: 0.356319 | norm_perceptual_loss: 29.711077 [00:24:52.588877] [Epoch 2] | Iter: 61900 | Optim: 61900 | Time: 0.77s | LR: 0.000002 | SkipOpt: 0 loss: 0.158495 | l2_loss: 0.009114 | psnr: 20.402885 | perceptual_loss: 0.298761 | norm_perceptual_loss: 32.780270 [00:25:51.138041] [Epoch 2] | Iter: 61950 | Optim: 61950 | Time: 1.27s | LR: 0.000002 | SkipOpt: 0 loss: 0.179231 | l2_loss: 0.010746 | psnr: 19.687504 | perceptual_loss: 0.336969 | norm_perceptual_loss: 31.357435 [00:26:52.602728] [Epoch 2] | Iter: 62000 | Optim: 62000 | Time: 1.91s | LR: 0.000002 | SkipOpt: 0 loss: 0.202464 | l2_loss: 0.013061 | psnr: 18.840258 | perceptual_loss: 0.378805 | norm_perceptual_loss: 29.002935 [00:26:59.060027] Saved checkpoint at iteration 62000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [00:26:59.060394] Saved latest checkpoint at iteration 62000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [00:27:58.678717] [Epoch 2] | Iter: 62050 | Optim: 62050 | Time: 1.91s | LR: 0.000002 | SkipOpt: 0 loss: 0.203775 | l2_loss: 0.012046 | psnr: 19.191525 | perceptual_loss: 0.383458 | norm_perceptual_loss: 31.832420 [00:29:00.913457] [Epoch 2] | Iter: 62100 | Optim: 62100 | Time: 1.08s | LR: 0.000002 | SkipOpt: 0 loss: 0.164168 | l2_loss: 0.007590 | psnr: 21.197512 | perceptual_loss: 0.313155 | norm_perceptual_loss: 41.258198 [00:30:08.151029] [Epoch 2] | Iter: 62150 | Optim: 62150 | Time: 0.75s | LR: 0.000002 | SkipOpt: 0 loss: 0.154518 | l2_loss: 0.008881 | psnr: 20.515232 | perceptual_loss: 0.291274 | norm_perceptual_loss: 32.796261 [00:31:08.752730] [Epoch 2] | Iter: 62200 | Optim: 62200 | Time: 1.07s | LR: 0.000002 | SkipOpt: 0 loss: 0.148091 | l2_loss: 0.005832 | psnr: 22.342022 | perceptual_loss: 0.284518 | norm_perceptual_loss: 48.787941 [00:32:16.353067] [Epoch 2] | Iter: 62250 | Optim: 62250 | Time: 1.91s | LR: 0.000002 | SkipOpt: 0 loss: 0.188925 | l2_loss: 0.013359 | psnr: 18.742254 | perceptual_loss: 0.351133 | norm_perceptual_loss: 26.284296 [00:33:16.823060] [Epoch 2] | Iter: 62300 | Optim: 62300 | Time: 1.91s | LR: 0.000002 | SkipOpt: 0 loss: 0.206930 | l2_loss: 0.014501 | psnr: 18.386068 | perceptual_loss: 0.384859 | norm_perceptual_loss: 26.540455 [00:34:24.620123] [Epoch 2] | Iter: 62350 | Optim: 62350 | Time: 1.07s | LR: 0.000002 | SkipOpt: 0 loss: 0.135567 | l2_loss: 0.008095 | psnr: 20.917929 | perceptual_loss: 0.254943 | norm_perceptual_loss: 31.494635 [00:35:26.806193] [Epoch 2] | Iter: 62400 | Optim: 62400 | Time: 1.91s | LR: 0.000002 | SkipOpt: 0 loss: 0.188561 | l2_loss: 0.011637 | psnr: 19.341639 | perceptual_loss: 0.353848 | norm_perceptual_loss: 30.407471 [00:36:26.157318] [Epoch 2] | Iter: 62450 | Optim: 62450 | Time: 1.08s | LR: 0.000002 | SkipOpt: 0 loss: 0.175443 | l2_loss: 0.011081 | psnr: 19.554180 | perceptual_loss: 0.328723 | norm_perceptual_loss: 29.665270 [00:37:22.354221] [Epoch 2] | Iter: 62500 | Optim: 62500 | Time: 1.27s | LR: 0.000002 | SkipOpt: 0 loss: 0.191726 | l2_loss: 0.012652 | psnr: 18.978277 | perceptual_loss: 0.358146 | norm_perceptual_loss: 28.306625 [00:38:26.697169] [Epoch 2] | Iter: 62550 | Optim: 62550 | Time: 1.08s | LR: 0.000002 | SkipOpt: 0 loss: 0.189760 | l2_loss: 0.011181 | psnr: 19.515221 | perceptual_loss: 0.357158 | norm_perceptual_loss: 31.943510 [00:39:31.825422] [Epoch 2] | Iter: 62600 | Optim: 62600 | Time: 1.91s | LR: 0.000001 | SkipOpt: 0 loss: 0.185779 | l2_loss: 0.011562 | psnr: 19.369839 | perceptual_loss: 0.348436 | norm_perceptual_loss: 30.137434 [00:40:31.196755] [Epoch 2] | Iter: 62650 | Optim: 62650 | Time: 1.27s | LR: 0.000001 | SkipOpt: 0 loss: 0.165125 | l2_loss: 0.008147 | psnr: 20.890251 | perceptual_loss: 0.313956 | norm_perceptual_loss: 38.538387 [00:41:27.209465] [Epoch 2] | Iter: 62700 | Optim: 62700 | Time: 0.75s | LR: 0.000001 | SkipOpt: 0 loss: 0.164869 | l2_loss: 0.009162 | psnr: 20.380241 | perceptual_loss: 0.311415 | norm_perceptual_loss: 33.991009 [00:42:29.667993] [Epoch 2] | Iter: 62750 | Optim: 62750 | Time: 1.91s | LR: 0.000001 | SkipOpt: 0 loss: 0.202667 | l2_loss: 0.012912 | psnr: 18.890013 | perceptual_loss: 0.379510 | norm_perceptual_loss: 29.391674 [00:43:46.426976] [Epoch 2] | Iter: 62800 | Optim: 62800 | Time: 0.75s | LR: 0.000001 | SkipOpt: 0 loss: 0.179006 | l2_loss: 0.008564 | psnr: 20.673197 | perceptual_loss: 0.340884 | norm_perceptual_loss: 39.803974 [00:44:49.384048] [Epoch 2] | Iter: 62850 | Optim: 62850 | Time: 0.75s | LR: 0.000001 | SkipOpt: 0 loss: 0.154367 | l2_loss: 0.007979 | psnr: 20.980310 | perceptual_loss: 0.292776 | norm_perceptual_loss: 36.691536 [00:45:54.261874] [Epoch 2] | Iter: 62900 | Optim: 62900 | Time: 0.75s | LR: 0.000001 | SkipOpt: 0 loss: 0.119846 | l2_loss: 0.003783 | psnr: 24.222029 | perceptual_loss: 0.232128 | norm_perceptual_loss: 61.366264 [00:46:56.088230] [Epoch 2] | Iter: 62950 | Optim: 62950 | Time: 0.85s | LR: 0.000001 | SkipOpt: 0 loss: 0.202510 | l2_loss: 0.016650 | psnr: 17.785948 | perceptual_loss: 0.371721 | norm_perceptual_loss: 22.326067 [00:47:36.579134] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [00:48:00.558579] [Epoch 2] | Iter: 63000 | Optim: 63000 | Time: 0.85s | LR: 0.000001 | SkipOpt: 0 loss: 0.220530 | l2_loss: 0.017226 | psnr: 17.638237 | perceptual_loss: 0.406609 | norm_perceptual_loss: 23.604807 [00:48:07.382960] Saved checkpoint at iteration 63000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [00:48:07.383403] Saved latest checkpoint at iteration 63000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [00:49:04.199422] [Epoch 2] | Iter: 63050 | Optim: 63050 | Time: 1.08s | LR: 0.000001 | SkipOpt: 0 loss: 0.135661 | l2_loss: 0.007319 | psnr: 21.355259 | perceptual_loss: 0.256683 | norm_perceptual_loss: 35.068962 [00:50:03.143441] [Epoch 2] | Iter: 63100 | Optim: 63100 | Time: 1.91s | LR: 0.000001 | SkipOpt: 0 loss: 0.194742 | l2_loss: 0.014735 | psnr: 18.316559 | perceptual_loss: 0.360014 | norm_perceptual_loss: 24.432884 [00:51:06.531448] [Epoch 2] | Iter: 63150 | Optim: 63150 | Time: 1.07s | LR: 0.000001 | SkipOpt: 0 loss: 0.188443 | l2_loss: 0.010661 | psnr: 19.722004 | perceptual_loss: 0.355565 | norm_perceptual_loss: 33.351784 [00:52:09.077654] [Epoch 2] | Iter: 63200 | Optim: 63200 | Time: 0.75s | LR: 0.000001 | SkipOpt: 0 loss: 0.152759 | l2_loss: 0.006371 | psnr: 21.957682 | perceptual_loss: 0.292776 | norm_perceptual_loss: 45.951908 [00:53:09.711694] [Epoch 2] | Iter: 63250 | Optim: 63250 | Time: 1.91s | LR: 0.000001 | SkipOpt: 0 loss: 0.165699 | l2_loss: 0.010127 | psnr: 19.945177 | perceptual_loss: 0.311145 | norm_perceptual_loss: 30.724140 [00:54:09.501201] [Epoch 2] | Iter: 63300 | Optim: 63300 | Time: 0.75s | LR: 0.000001 | SkipOpt: 0 loss: 0.163763 | l2_loss: 0.009031 | psnr: 20.442457 | perceptual_loss: 0.309463 | norm_perceptual_loss: 34.265247 [00:55:13.854120] [Epoch 2] | Iter: 63350 | Optim: 63350 | Time: 0.75s | LR: 0.000001 | SkipOpt: 0 loss: 0.173202 | l2_loss: 0.009673 | psnr: 20.144432 | perceptual_loss: 0.327059 | norm_perceptual_loss: 33.811882 [00:56:08.981226] New dataloader... [00:56:08.982092] Building dynamic dataloader with seed: 0 [00:56:09.059818] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.059979] Worker seed: 37036 [00:56:09.112034] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.112180] Worker seed: 37037 [00:56:09.163681] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.163819] Worker seed: 37038 [00:56:09.201597] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.201744] Worker seed: 37039 [00:56:09.238561] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.238701] Worker seed: 37040 [00:56:09.274126] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.274272] Worker seed: 37041 [00:56:09.309843] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.309988] Worker seed: 37042 [00:56:09.344674] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.344812] Worker seed: 37043 [00:56:09.379723] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.379865] Worker seed: 37044 [00:56:09.414334] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.414479] Worker seed: 37045 [00:56:09.448595] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.448726] Worker seed: 37046 [00:56:09.482472] Rank: 0, World size: 1, Distributed rank: 0 [00:56:09.482613] Worker seed: 37047 [00:56:19.069268] [Epoch 3] | Iter: 63400 | Optim: 63400 | Time: 1.07s | LR: 0.000001 | SkipOpt: 0 loss: 0.150631 | l2_loss: 0.008374 | psnr: 20.770781 | perceptual_loss: 0.284515 | norm_perceptual_loss: 33.976826 [00:57:14.494322] [Epoch 3] | Iter: 63450 | Optim: 63450 | Time: 0.75s | LR: 0.000001 | SkipOpt: 0 loss: 0.163219 | l2_loss: 0.007443 | psnr: 21.282656 | perceptual_loss: 0.311553 | norm_perceptual_loss: 41.859783 [00:58:18.190936] [Epoch 3] | Iter: 63500 | Optim: 63500 | Time: 1.08s | LR: 0.000001 | SkipOpt: 0 loss: 0.225997 | l2_loss: 0.018470 | psnr: 17.335239 | perceptual_loss: 0.415053 | norm_perceptual_loss: 22.471231 [00:58:35.420236] seq 2K/d26afc876a34e53738d5970d071134008b059c6a1c4b605622be621063c7e387 returned frame_indices None - has only 10 images [00:59:12.465760] [Epoch 3] | Iter: 63550 | Optim: 63550 | Time: 1.91s | LR: 0.000001 | SkipOpt: 0 loss: 0.155109 | l2_loss: 0.008476 | psnr: 20.718315 | perceptual_loss: 0.293266 | norm_perceptual_loss: 34.601391 [01:00:16.878683] [Epoch 3] | Iter: 63600 | Optim: 63600 | Time: 1.91s | LR: 0.000001 | SkipOpt: 0 loss: 0.189428 | l2_loss: 0.014378 | psnr: 18.423012 | perceptual_loss: 0.350099 | norm_perceptual_loss: 24.349621 [01:01:14.010365] [Epoch 3] | Iter: 63650 | Optim: 63650 | Time: 1.92s | LR: 0.000000 | SkipOpt: 0 loss: 0.187452 | l2_loss: 0.012444 | psnr: 19.050518 | perceptual_loss: 0.350017 | norm_perceptual_loss: 28.128149 [01:02:16.719955] [Epoch 3] | Iter: 63700 | Optim: 63700 | Time: 0.76s | LR: 0.000000 | SkipOpt: 0 loss: 0.153495 | l2_loss: 0.009038 | psnr: 20.439240 | perceptual_loss: 0.288913 | norm_perceptual_loss: 31.966185 [01:03:19.118949] [Epoch 3] | Iter: 63750 | Optim: 63750 | Time: 1.07s | LR: 0.000000 | SkipOpt: 0 loss: 0.151114 | l2_loss: 0.007909 | psnr: 21.018944 | perceptual_loss: 0.286411 | norm_perceptual_loss: 36.214691 [01:04:19.745774] [Epoch 3] | Iter: 63800 | Optim: 63800 | Time: 1.91s | LR: 0.000000 | SkipOpt: 0 loss: 0.199182 | l2_loss: 0.012033 | psnr: 19.196114 | perceptual_loss: 0.374298 | norm_perceptual_loss: 31.104923 [01:05:23.530738] [Epoch 3] | Iter: 63850 | Optim: 63850 | Time: 1.91s | LR: 0.000000 | SkipOpt: 0 loss: 0.184095 | l2_loss: 0.013555 | psnr: 18.678972 | perceptual_loss: 0.341080 | norm_perceptual_loss: 25.162497 [01:06:19.169664] [Epoch 3] | Iter: 63900 | Optim: 63900 | Time: 0.76s | LR: 0.000000 | SkipOpt: 0 loss: 0.143628 | l2_loss: 0.005400 | psnr: 22.676224 | perceptual_loss: 0.276456 | norm_perceptual_loss: 51.197430 [01:07:22.876483] [Epoch 3] | Iter: 63950 | Optim: 63950 | Time: 0.75s | LR: 0.000000 | SkipOpt: 0 loss: 0.136570 | l2_loss: 0.007514 | psnr: 21.241018 | perceptual_loss: 0.258111 | norm_perceptual_loss: 34.348488 [01:08:23.191352] [Epoch 3] | Iter: 64000 | Optim: 64000 | Time: 1.27s | LR: 0.000000 | SkipOpt: 0 loss: 0.190998 | l2_loss: 0.012759 | psnr: 18.941710 | perceptual_loss: 0.356478 | norm_perceptual_loss: 27.938553 [01:08:28.864337] Saved checkpoint at iteration 64000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [01:08:28.864709] Saved latest checkpoint at iteration 64000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [01:09:25.917955] [Epoch 3] | Iter: 64050 | Optim: 64050 | Time: 0.85s | LR: 0.000000 | SkipOpt: 0 loss: 0.153779 | l2_loss: 0.007971 | psnr: 20.984680 | perceptual_loss: 0.291615 | norm_perceptual_loss: 36.582851 [01:10:23.106508] [Epoch 3] | Iter: 64100 | Optim: 64100 | Time: 1.07s | LR: 0.000000 | SkipOpt: 0 loss: 0.154183 | l2_loss: 0.008942 | psnr: 20.485716 | perceptual_loss: 0.290482 | norm_perceptual_loss: 32.485550 [01:11:24.620100] [Epoch 3] | Iter: 64150 | Optim: 64150 | Time: 1.08s | LR: 0.000000 | SkipOpt: 0 loss: 0.188727 | l2_loss: 0.009007 | psnr: 20.454103 | perceptual_loss: 0.359440 | norm_perceptual_loss: 39.905876 [01:12:31.077370] [Epoch 3] | Iter: 64200 | Optim: 64200 | Time: 1.08s | LR: 0.000000 | SkipOpt: 0 loss: 0.161679 | l2_loss: 0.007791 | psnr: 21.083797 | perceptual_loss: 0.307774 | norm_perceptual_loss: 39.501362 [01:13:39.143808] [Epoch 3] | Iter: 64250 | Optim: 64250 | Time: 1.07s | LR: 0.000000 | SkipOpt: 0 loss: 0.199223 | l2_loss: 0.015306 | psnr: 18.151466 | perceptual_loss: 0.367834 | norm_perceptual_loss: 24.032455 [01:14:35.276891] [Epoch 3] | Iter: 64300 | Optim: 64300 | Time: 1.08s | LR: 0.000000 | SkipOpt: 0 loss: 0.167590 | l2_loss: 0.011264 | psnr: 19.483019 | perceptual_loss: 0.312651 | norm_perceptual_loss: 27.756317 [01:15:43.123400] [Epoch 3] | Iter: 64350 | Optim: 64350 | Time: 1.91s | LR: 0.000000 | SkipOpt: 0 loss: 0.199088 | l2_loss: 0.015388 | psnr: 18.128078 | perceptual_loss: 0.367398 | norm_perceptual_loss: 23.875092 [01:16:48.767616] [Epoch 3] | Iter: 64400 | Optim: 64400 | Time: 0.75s | LR: 0.000000 | SkipOpt: 0 loss: 0.164842 | l2_loss: 0.009909 | psnr: 20.039843 | perceptual_loss: 0.309867 | norm_perceptual_loss: 31.272287 [01:17:49.865798] [Epoch 3] | Iter: 64450 | Optim: 64450 | Time: 1.07s | LR: 0.000000 | SkipOpt: 0 loss: 0.164341 | l2_loss: 0.008784 | psnr: 20.563288 | perceptual_loss: 0.311114 | norm_perceptual_loss: 35.419971 [01:18:51.881108] [Epoch 3] | Iter: 64500 | Optim: 64500 | Time: 1.92s | LR: 0.000000 | SkipOpt: 0 loss: 0.174912 | l2_loss: 0.009715 | psnr: 20.125713 | perceptual_loss: 0.330395 | norm_perceptual_loss: 34.009842 [01:19:59.055301] [Epoch 3] | Iter: 64550 | Optim: 64550 | Time: 0.75s | LR: 0.000000 | SkipOpt: 0 loss: 0.167264 | l2_loss: 0.007953 | psnr: 20.994801 | perceptual_loss: 0.318623 | norm_perceptual_loss: 40.064316 [01:21:03.466587] [Epoch 3] | Iter: 64600 | Optim: 64600 | Time: 1.91s | LR: 0.000000 | SkipOpt: 0 loss: 0.191791 | l2_loss: 0.011071 | psnr: 19.558090 | perceptual_loss: 0.361441 | norm_perceptual_loss: 32.647209 [01:22:05.253585] [Epoch 3] | Iter: 64650 | Optim: 64650 | Time: 0.77s | LR: 0.000000 | SkipOpt: 0 loss: 0.126652 | l2_loss: 0.006473 | psnr: 21.888912 | perceptual_loss: 0.240357 | norm_perceptual_loss: 37.131985 [01:23:05.552409] [Epoch 3] | Iter: 64700 | Optim: 64700 | Time: 1.07s | LR: 0.000000 | SkipOpt: 0 loss: 0.183842 | l2_loss: 0.012699 | psnr: 18.962254 | perceptual_loss: 0.342286 | norm_perceptual_loss: 26.953485 [01:24:07.181902] [Epoch 3] | Iter: 64750 | Optim: 64750 | Time: 1.91s | LR: 0.000000 | SkipOpt: 0 loss: 0.167747 | l2_loss: 0.008892 | psnr: 20.509989 | perceptual_loss: 0.317711 | norm_perceptual_loss: 35.729809 [01:25:07.770189] [Epoch 3] | Iter: 64800 | Optim: 64800 | Time: 1.07s | LR: 0.000000 | SkipOpt: 0 loss: 0.176459 | l2_loss: 0.011016 | psnr: 19.579605 | perceptual_loss: 0.330886 | norm_perceptual_loss: 30.035782 [01:26:15.301840] [Epoch 3] | Iter: 64850 | Optim: 64850 | Time: 1.91s | LR: 0.000000 | SkipOpt: 0 loss: 0.161873 | l2_loss: 0.011357 | psnr: 19.447275 | perceptual_loss: 0.301031 | norm_perceptual_loss: 26.505627 [01:27:19.128252] [Epoch 3] | Iter: 64900 | Optim: 64900 | Time: 1.07s | LR: 0.000000 | SkipOpt: 0 loss: 0.174589 | l2_loss: 0.010864 | psnr: 19.639935 | perceptual_loss: 0.327449 | norm_perceptual_loss: 30.139559 [01:28:16.046727] [Epoch 3] | Iter: 64950 | Optim: 64950 | Time: 0.75s | LR: 0.000000 | SkipOpt: 0 loss: 0.165844 | l2_loss: 0.009876 | psnr: 20.054146 | perceptual_loss: 0.311935 | norm_perceptual_loss: 31.584866 [01:29:16.596095] [Epoch 3] | Iter: 65000 | Optim: 65000 | Time: 1.07s | LR: 0.000000 | SkipOpt: 0 loss: 0.180924 | l2_loss: 0.008583 | psnr: 20.663803 | perceptual_loss: 0.344682 | norm_perceptual_loss: 40.160511 [01:29:18.400512] Found 140 sequences [01:29:41.606334] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [01:29:42.818956] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [01:30:34.369555] Gathering scores from all processes... [01:30:34.370393] Padding local tensor of length 140 to max count: 140 [01:30:34.370971] Padding local tensor of length 140 to max count: 140 [01:30:34.371517] Padding local tensor of length 140 to max count: 140 [01:30:34.371832] Gathering scene names from all processes... [01:30:34.371982] Padding local tensor of length 140 on rank 0 to max count: 140 [01:30:34.373176] Evaluation metrics: [01:30:34.373386] psnr: 17.6493 [01:30:34.373494] ssim: 0.5039 [01:30:34.373595] lpips: 0.4750 [01:30:34.377294] Found 140 sequences [01:30:57.972387] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [01:30:59.109978] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [01:31:52.019436] Gathering scores from all processes... [01:31:52.020511] Padding local tensor of length 140 to max count: 140 [01:31:52.021321] Padding local tensor of length 140 to max count: 140 [01:31:52.021822] Padding local tensor of length 140 to max count: 140 [01:31:52.022142] Gathering scene names from all processes... [01:31:52.022252] Padding local tensor of length 140 on rank 0 to max count: 140 [01:31:52.023348] Evaluation metrics: [01:31:52.023653] psnr: 20.6843 [01:31:52.023814] ssim: 0.6070 [01:31:52.023971] lpips: 0.3581 [01:31:52.029038] Found 140 sequences [01:32:15.327215] Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] [01:32:16.266404] Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/vgg.pth [01:33:12.015373] Gathering scores from all processes... [01:33:12.016190] Padding local tensor of length 140 to max count: 140 [01:33:12.016766] Padding local tensor of length 140 to max count: 140 [01:33:12.017240] Padding local tensor of length 140 to max count: 140 [01:33:12.017502] Gathering scene names from all processes... [01:33:12.017615] Padding local tensor of length 140 on rank 0 to max count: 140 [01:33:12.018566] Evaluation metrics: [01:33:12.018726] psnr: 21.7782 [01:33:12.018810] ssim: 0.6443 [01:33:12.018882] lpips: 0.3197 [01:33:12.020286] [Eval][iter 65000] dl3dv_2view | psnr: 17.649261 | ssim: 0.503940 | lpips: 0.474991 || dl3dv_4view | psnr: 20.684305 | ssim: 0.607008 | lpips: 0.358075 || dl3dv_6view | psnr: 21.778204 | ssim: 0.644274 | lpips: 0.319716 [01:33:12.023187] [Eval][iter 65000] dl3dv_2view | psnr: 17.649261 | ssim: 0.503940 | lpips: 0.474991 || dl3dv_4view | psnr: 20.684305 | ssim: 0.607008 | lpips: 0.358075 || dl3dv_6view | psnr: 21.778204 | ssim: 0.644274 | lpips: 0.319716 [01:33:16.174775] Saved checkpoint at iteration 65000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [01:33:16.175268] Saved latest checkpoint at iteration 65000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [01:33:19.910449] Saved checkpoint at iteration 65000: ./output/abl_lact_l24/checkpoints/checkpoint_0065000.pt [01:33:19.910814] Saved checkpoint at iteration 65000 to ./output/abl_lact_l24/checkpoints/checkpoint_0065000.pt [01:33:25.045215] Saved checkpoint at iteration 65000: ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt [01:33:25.046140] Saved latest checkpoint at iteration 65000 to ./output/abl_lact_l24/checkpoints/checkpoint_latest.pt wandb: - 231.641 MB of 231.641 MB uploaded wandb: \ 231.999 MB of 231.999 MB uploaded wandb: | 232.004 MB of 232.004 MB uploaded wandb: / 232.004 MB of 232.004 MB uploaded wandb: wandb: wandb: Run history: wandb: epoch ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▆▆▆▆▆▆▆▆▆█ wandb: eval/dl3dv_2view/lpips █▅▄▃▃▂▂▂▁▁▁▁▁▁ wandb: eval/dl3dv_2view/psnr ▁▆▇▇▇▇▇▇██████ wandb: eval/dl3dv_2view/ssim ▁▇▇▇▇▇▇███████ wandb: eval/dl3dv_4view/lpips █▆▄▃▃▃▂▂▂▁▁▁▁▁ wandb: eval/dl3dv_4view/psnr ▁▆▆▆▇▇▇▇▇█████ wandb: eval/dl3dv_4view/ssim ▁▆▆▇▇▇▇▇██████ wandb: eval/dl3dv_6view/lpips █▆▄▃▃▃▂▂▂▁▁▁▁▁ wandb: eval/dl3dv_6view/psnr ▁▆▆▆▇▇▇▇▇█████ wandb: eval/dl3dv_6view/ssim ▁▆▆▇▇▇▇▇██████ wandb: iter ▁▁▁▁▁▁▂▂▂▂▂▂▂▃▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇█ wandb: iter_time ███▂▁▁█▄██▃█▁█▃▁▁▁▃▁▁█▃▃█▄█▁▁██▃██▁▃▃▁▁█ wandb: lr ▁▁▁▁▁▁▁▁█████▇▇▇▇▇▆▆▆▆▅▅▅▅▄▄▃▂▂▂▂▁▁▁▁▁▁▁ wandb: lr_encoder ▁▁▁▁▁▁▁▃▃▇██████▇▇▇▇▅▅▅▅▄▄▄▄▃▃▃▃▂▂▂▂▂▂▁▁ wandb: skip_optimizer_step ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ wandb: train/l2_loss ▇▆▆▆▆█▅▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▂▁▂▁▂▂▂▁▂▂▂▂▁ wandb: train/loss ▆██▇█▇▇▆▄▄▄▄▃▄▄▄▄▃▃▃▂▂▂▂▂▂▃▃▂▃▂▃▃▁▃▁▂▂▃▁ wandb: train/norm_perceptual_loss ▁▁▁▁▃▄▄▄▄▆▄▅▅▆▄▄▄▅▃▃▄▅▄▅▄▅▄▄▆▇▄▅▄▅▅▄▅█▅▃ wandb: train/perceptual_loss ▇█▇▇▅▅▄▄▄▃▄▃▃▄▂▂▂▃▃▃▃▃▄▂▂▂▃▂▃▂▁▃▁▂▁▃▃▁▃▃ wandb: train/psnr ▁▁▂▁▁▂▄▃▄▅▅▇▆▆▄▇▅▅▅▅▆▅▆▆▇▆▆▆▅▅▆▅▆▆▇▆█▆▅▇ wandb: wandb: Run summary: wandb: epoch 3 wandb: eval/dl3dv_2view/lpips 0.47499 wandb: eval/dl3dv_2view/psnr 17.64926 wandb: eval/dl3dv_2view/ssim 0.50394 wandb: eval/dl3dv_4view/lpips 0.35807 wandb: eval/dl3dv_4view/psnr 20.68431 wandb: eval/dl3dv_4view/ssim 0.60701 wandb: eval/dl3dv_6view/lpips 0.31972 wandb: eval/dl3dv_6view/psnr 21.7782 wandb: eval/dl3dv_6view/ssim 0.64427 wandb: iter 65000 wandb: iter_time 1.07074 wandb: lr 0 wandb: lr_encoder 0 wandb: skip_optimizer_step 0 wandb: train/l2_loss 0.01192 wandb: train/loss 0.16851 wandb: train/norm_perceptual_loss 26.2742 wandb: train/perceptual_loss 0.31318 wandb: train/psnr 19.23735 wandb: wandb: 🚀 View run abl_lact_l24 at: https://wandb.ai/dudrms514/abl_enc6/runs/i2mao1ll wandb: ⭐️ View project at: https://wandb.ai/dudrms514/abl_enc6 wandb: Synced 4 W&B file(s), 0 media file(s), 8 artifact file(s) and 131 other file(s) wandb: Find logs at: ./output/abl_lact_l24/checkpoints/wandb/run-20261113_014747-i2mao1ll/logs [rank0]:[W1114 01:33:35.381660290 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())