diff --git "a/training.log" "b/training.log" new file mode 100644--- /dev/null +++ "b/training.log" @@ -0,0 +1,38622 @@ +[2026-08-26 12:35:25,867] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:35:26,008] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:35:26,104] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:35:26,224] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:35:26,233] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:35:26,259] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:35:26,260] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:35:26,313] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:35:26,366] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:35:26,386] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:35:27,036] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:35:27,690] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:35:27,690] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl +[2026-08-26 12:35:27,726] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:35:27,743] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:35:27,762] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:35:27,794] [INFO] [comm.py:652:init_distributed] cdb=None +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) +08/26/2026 12:35:54 - INFO - root - Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) +08/26/2026 12:35:54 - INFO - root - Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loaded 977344 samples +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Force set image size to be 448 +08/26/2026 12:36:07 - INFO - root - Force set image size to be 448 +08/26/2026 12:36:07 - INFO - root - Force set image size to be 448 +[INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) +08/26/2026 12:36:07 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) +08/26/2026 12:36:07 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) +Use meta file to control datasets loading. Data path will use as meta path +08/26/2026 12:36:07 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path +08/26/2026 12:36:07 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +[rank3]: Traceback (most recent call last): +[rank3]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in +[rank3]: train(attn_implementation="flash_attention_2") +[rank3]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train +[rank3]: trainer = VideoLLaMA3Trainer( +[rank3]: ^^^^^^^^^^^^^^^^^^^ +[rank3]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ +[rank3]: super().__init__(*args, **kwargs) +[rank3]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func +[rank3]: return func(*args, **kwargs) +[rank3]: ^^^^^^^^^^^^^^^^^^^^^ +[rank3]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ +[rank3]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) +[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +[rank3]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks +[rank3]: raise ValueError( +[rank3]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +[rank0]: Traceback (most recent call last): +[rank0]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in +[rank0]: train(attn_implementation="flash_attention_2") +[rank0]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train +[rank0]: trainer = VideoLLaMA3Trainer( +[rank0]: ^^^^^^^^^^^^^^^^^^^ +[rank0]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ +[rank0]: super().__init__(*args, **kwargs) +[rank0]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func +[rank0]: return func(*args, **kwargs) +[rank0]: ^^^^^^^^^^^^^^^^^^^^^ +[rank0]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ +[rank0]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) +[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +[rank0]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks +[rank0]: raise ValueError( +[rank0]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +[rank5]: Traceback (most recent call last): +[rank5]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in +[rank5]: train(attn_implementation="flash_attention_2") +[rank5]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train +[rank5]: trainer = VideoLLaMA3Trainer( +[rank5]: ^^^^^^^^^^^^^^^^^^^ +[rank5]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ +[rank5]: super().__init__(*args, **kwargs) +[rank5]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func +[rank5]: return func(*args, **kwargs) +[rank5]: ^^^^^^^^^^^^^^^^^^^^^ +[rank5]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ +[rank5]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) +[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +[rank5]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks +[rank5]: raise ValueError( +[rank5]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +[rank6]: Traceback (most recent call last): +[rank6]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in +[rank6]: train(attn_implementation="flash_attention_2") +[rank6]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train +[rank6]: trainer = VideoLLaMA3Trainer( +[rank6]: ^^^^^^^^^^^^^^^^^^^ +[rank6]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ +[rank6]: super().__init__(*args, **kwargs) +[rank6]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func +[rank6]: return func(*args, **kwargs) +[rank6]: ^^^^^^^^^^^^^^^^^^^^^ +[rank6]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ +[rank6]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) +[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +[rank6]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks +[rank6]: raise ValueError( +[rank6]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +[rank1]: Traceback (most recent call last): +[rank1]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in +[rank1]: train(attn_implementation="flash_attention_2") +[rank1]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train +[rank1]: trainer = VideoLLaMA3Trainer( +[rank1]: ^^^^^^^^^^^^^^^^^^^ +[rank1]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ +[rank1]: super().__init__(*args, **kwargs) +[rank1]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func +[rank1]: return func(*args, **kwargs) +[rank1]: ^^^^^^^^^^^^^^^^^^^^^ +[rank1]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ +[rank1]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) +[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +[rank1]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks +[rank1]: raise ValueError( +[rank1]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +[rank2]: Traceback (most recent call last): +[rank2]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in +[rank2]: train(attn_implementation="flash_attention_2") +[rank2]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train +[rank2]: trainer = VideoLLaMA3Trainer( +[rank2]: ^^^^^^^^^^^^^^^^^^^ +[rank2]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ +[rank2]: super().__init__(*args, **kwargs) +[rank2]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func +[rank2]: return func(*args, **kwargs) +[rank2]: ^^^^^^^^^^^^^^^^^^^^^ +[rank2]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ +[rank2]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) +[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +[rank2]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks +[rank2]: raise ValueError( +[rank2]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. +[rank0]:[W826 12:37:41.854100677 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +[rank7]: Traceback (most recent call last): +[rank7]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in +[rank7]: train(attn_implementation="flash_attention_2") +[rank7]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train +[rank7]: trainer = VideoLLaMA3Trainer( +[rank7]: ^^^^^^^^^^^^^^^^^^^ +[rank7]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ +[rank7]: super().__init__(*args, **kwargs) +[rank7]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func +[rank7]: return func(*args, **kwargs) +[rank7]: ^^^^^^^^^^^^^^^^^^^^^ +[rank7]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ +[rank7]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) +[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +[rank7]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks +[rank7]: raise ValueError( +[rank7]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. +W0826 12:37:41.291000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229306 closing signal SIGTERM +W0826 12:37:41.292000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229307 closing signal SIGTERM +W0826 12:37:41.293000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229308 closing signal SIGTERM +W0826 12:37:41.295000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229310 closing signal SIGTERM +W0826 12:37:41.296000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229311 closing signal SIGTERM +W0826 12:37:41.297000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229312 closing signal SIGTERM +W0826 12:37:41.299000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229313 closing signal SIGTERM +E0826 12:37:42.220000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 3 (pid: 229309) of binary: /miniconda/envs/ml/bin/python3.12 +Traceback (most recent call last): + File "/miniconda/envs/ml/bin/torchrun", line 6, in + sys.exit(main()) + ^^^^^^ + File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper + return f(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^ + File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/run.py", line 936, in main + run(args) + File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/run.py", line 927, in run + elastic_launch( + File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 156, in __call__ + return launch_agent(self._config, self._entrypoint, list(args)) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent + raise ChildFailedError( +torch.distributed.elastic.multiprocessing.errors.ChildFailedError: +============================================================ +videollama3/train/compressor_pretrain_with_videollama3.py FAILED +------------------------------------------------------------ +Failures: + +------------------------------------------------------------ +Root Cause (first observed failure): +[0]: + time : 2026-08-26_12:37:41 + host : ce-5a549e89-ba00-4667-9c22-ea966823c3fe-7bb99844bc-f72hx + rank : 3 (local_rank: 3) + exitcode : 1 (pid: 229309) + error_file: + traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html +============================================================ +[2026-08-26 12:39:37,885] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:39:37,962] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:39:38,014] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:39:38,287] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:39:38,373] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:39:38,374] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:39:38,406] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:39:38,417] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:39:38,468] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:39:38,477] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:39:38,505] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2026-08-26 12:39:39,127] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:39:39,224] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:39:39,229] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:39:39,234] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:39:39,252] [INFO] [comm.py:652:init_distributed] cdb=None +[2026-08-26 12:39:39,252] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +`torch_dtype` is deprecated! Use `dtype` instead! +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. +Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) +08/26/2026 12:40:07 - INFO - root - Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) +08/26/2026 12:40:07 - INFO - root - Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Loaded 977344 samples +Force set image size to be 448 +08/26/2026 12:40:20 - INFO - root - Force set image size to be 448 +08/26/2026 12:40:20 - INFO - root - Force set image size to be 448 +[INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) +08/26/2026 12:40:20 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) +08/26/2026 12:40:20 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) +Use meta file to control datasets loading. Data path will use as meta path +08/26/2026 12:40:20 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path +08/26/2026 12:40:20 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path +Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache +Loading anno_online/detail_caption_recap.json via `load_dataset` +Loaded 977344 samples +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. +The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. +/root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. + super().__init__(*args, **kwargs) +The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. +wandb: Loading settings from /root/OnlineLong/wandb/settings +wandb: Tracking run with wandb version 0.28.2 +wandb: W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing. +wandb: Run data is saved locally in /root/OnlineLong/wandb/offline-run-20260826_124205-av1xzhof +wandb: View this run in the terminal with `wandb leet` + 0%| | 0/7636 [00:00