[2026-08-26 12:35:25,867] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:35:26,008] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:35:26,104] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:35:26,224] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:35:26,233] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:35:26,259] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:35:26,260] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:35:26,313] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:35:26,366] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:35:26,386] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:35:27,036] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:35:27,690] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:35:27,690] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2026-08-26 12:35:27,726] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:35:27,743] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:35:27,762] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:35:27,794] [INFO] [comm.py:652:init_distributed] cdb=None `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) 08/26/2026 12:35:54 - INFO - root - Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) 08/26/2026 12:35:54 - INFO - root - Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loaded 977344 samples Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Force set image size to be 448 08/26/2026 12:36:07 - INFO - root - Force set image size to be 448 08/26/2026 12:36:07 - INFO - root - Force set image size to be 448 [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) 08/26/2026 12:36:07 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) 08/26/2026 12:36:07 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) Use meta file to control datasets loading. Data path will use as meta path 08/26/2026 12:36:07 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path 08/26/2026 12:36:07 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) [rank3]: Traceback (most recent call last): [rank3]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in [rank3]: train(attn_implementation="flash_attention_2") [rank3]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train [rank3]: trainer = VideoLLaMA3Trainer( [rank3]: ^^^^^^^^^^^^^^^^^^^ [rank3]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ [rank3]: super().__init__(*args, **kwargs) [rank3]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func [rank3]: return func(*args, **kwargs) [rank3]: ^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ [rank3]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks [rank3]: raise ValueError( [rank3]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) [rank0]: Traceback (most recent call last): [rank0]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in [rank0]: train(attn_implementation="flash_attention_2") [rank0]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train [rank0]: trainer = VideoLLaMA3Trainer( [rank0]: ^^^^^^^^^^^^^^^^^^^ [rank0]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ [rank0]: super().__init__(*args, **kwargs) [rank0]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func [rank0]: return func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ [rank0]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks [rank0]: raise ValueError( [rank0]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) [rank5]: Traceback (most recent call last): [rank5]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in [rank5]: train(attn_implementation="flash_attention_2") [rank5]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train [rank5]: trainer = VideoLLaMA3Trainer( [rank5]: ^^^^^^^^^^^^^^^^^^^ [rank5]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ [rank5]: super().__init__(*args, **kwargs) [rank5]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func [rank5]: return func(*args, **kwargs) [rank5]: ^^^^^^^^^^^^^^^^^^^^^ [rank5]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ [rank5]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank5]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks [rank5]: raise ValueError( [rank5]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) [rank6]: Traceback (most recent call last): [rank6]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in [rank6]: train(attn_implementation="flash_attention_2") [rank6]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train [rank6]: trainer = VideoLLaMA3Trainer( [rank6]: ^^^^^^^^^^^^^^^^^^^ [rank6]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ [rank6]: super().__init__(*args, **kwargs) [rank6]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func [rank6]: return func(*args, **kwargs) [rank6]: ^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ [rank6]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks [rank6]: raise ValueError( [rank6]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) [rank1]: Traceback (most recent call last): [rank1]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in [rank1]: train(attn_implementation="flash_attention_2") [rank1]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train [rank1]: trainer = VideoLLaMA3Trainer( [rank1]: ^^^^^^^^^^^^^^^^^^^ [rank1]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ [rank1]: super().__init__(*args, **kwargs) [rank1]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func [rank1]: return func(*args, **kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ [rank1]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks [rank1]: raise ValueError( [rank1]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) [rank2]: Traceback (most recent call last): [rank2]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in [rank2]: train(attn_implementation="flash_attention_2") [rank2]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train [rank2]: trainer = VideoLLaMA3Trainer( [rank2]: ^^^^^^^^^^^^^^^^^^^ [rank2]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ [rank2]: super().__init__(*args, **kwargs) [rank2]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func [rank2]: return func(*args, **kwargs) [rank2]: ^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ [rank2]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks [rank2]: raise ValueError( [rank2]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. [rank0]:[W826 12:37:41.854100677 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) [rank7]: Traceback (most recent call last): [rank7]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 883, in [rank7]: train(attn_implementation="flash_attention_2") [rank7]: File "/root/OnlineLong/videollama3/train/compressor_pretrain_with_videollama3.py", line 850, in train [rank7]: trainer = VideoLLaMA3Trainer( [rank7]: ^^^^^^^^^^^^^^^^^^^ [rank7]: File "/root/OnlineLong/videollama3/train/videollama3_trainer.py", line 227, in __init__ [rank7]: super().__init__(*args, **kwargs) [rank7]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func [rank7]: return func(*args, **kwargs) [rank7]: ^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/trainer.py", line 678, in __init__ [rank7]: default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to) [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/miniconda/envs/ml/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2517, in get_reporting_integration_callbacks [rank7]: raise ValueError( [rank7]: ValueError: wandbs is not supported, only azure_ml, comet_ml, mlflow, neptune, tensorboard, trackio, wandb, codecarbon, clearml, dagshub, flyte, dvclive, swanlab are supported. W0826 12:37:41.291000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229306 closing signal SIGTERM W0826 12:37:41.292000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229307 closing signal SIGTERM W0826 12:37:41.293000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229308 closing signal SIGTERM W0826 12:37:41.295000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229310 closing signal SIGTERM W0826 12:37:41.296000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229311 closing signal SIGTERM W0826 12:37:41.297000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229312 closing signal SIGTERM W0826 12:37:41.299000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 229313 closing signal SIGTERM E0826 12:37:42.220000 229286 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 3 (pid: 229309) of binary: /miniconda/envs/ml/bin/python3.12 Traceback (most recent call last): File "/miniconda/envs/ml/bin/torchrun", line 6, in sys.exit(main()) ^^^^^^ File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/run.py", line 936, in main run(args) File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/run.py", line 927, in run elastic_launch( File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 156, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda/envs/ml/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ videollama3/train/compressor_pretrain_with_videollama3.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2026-08-26_12:37:41 host : ce-5a549e89-ba00-4667-9c22-ea966823c3fe-7bb99844bc-f72hx rank : 3 (local_rank: 3) exitcode : 1 (pid: 229309) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ [2026-08-26 12:39:37,885] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:39:37,962] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:39:38,014] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:39:38,287] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:39:38,373] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:39:38,374] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:39:38,406] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:39:38,417] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:39:38,468] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:39:38,477] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:39:38,505] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 12:39:39,127] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:39:39,224] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:39:39,229] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:39:39,234] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:39:39,252] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 12:39:39,252] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) 08/26/2026 12:40:07 - INFO - root - Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) 08/26/2026 12:40:07 - INFO - root - Trainable parameters: 715,262,976 / 8,215,010,544 (8.7068%) Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Force set image size to be 448 08/26/2026 12:40:20 - INFO - root - Force set image size to be 448 08/26/2026 12:40:20 - INFO - root - Force set image size to be 448 [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) 08/26/2026 12:40:20 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) 08/26/2026 12:40:20 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder, max_frames=8, fixed_frames=8) Use meta file to control datasets loading. Data path will use as meta path 08/26/2026 12:40:20 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path 08/26/2026 12:40:20 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loaded 977344 samples /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. wandb: Loading settings from /root/OnlineLong/wandb/settings wandb: Tracking run with wandb version 0.28.2 wandb: W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing. wandb: Run data is saved locally in /root/OnlineLong/wandb/offline-run-20260826_124205-av1xzhof wandb: View this run in the terminal with `wandb leet` 0%| | 0/7636 [00:00