[2026-08-26 19:32:19,807] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 19:32:19,834] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 19:32:19,835] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 19:32:19,836] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 19:32:19,843] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 19:32:19,860] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 19:32:19,873] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 19:32:19,963] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-08-26 19:32:20,597] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 19:32:20,597] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 19:32:20,597] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 19:32:20,598] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 19:32:20,603] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 19:32:20,605] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 19:32:20,605] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2026-08-26 19:32:20,605] [INFO] [comm.py:652:init_distributed] cdb=None [2026-08-26 19:32:20,606] [INFO] [comm.py:652:init_distributed] cdb=None `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. The tokenizer you are loading from 'pretrained_models/videollama3_7b_local' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. Trainable parameters: 715,004,928 / 8,214,752,496 (8.7039%) 08/26/2026 19:32:48 - INFO - root - Trainable parameters: 715,004,928 / 8,214,752,496 (8.7039%) 08/26/2026 19:32:48 - INFO - root - Trainable parameters: 715,004,928 / 8,214,752,496 (8.7039%) Force set image size to be 448 08/26/2026 19:33:00 - INFO - root - Force set image size to be 448 08/26/2026 19:33:00 - INFO - root - Force set image size to be 448 [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder_flat, max_frames=8, fixed_frames=8) 08/26/2026 19:33:00 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder_flat, max_frames=8, fixed_frames=8) 08/26/2026 19:33:00 - INFO - root - [INFO] Whole-video compression: 1 window per sample -> 256 tokens (compressor_type=transformer_decoder_flat, max_frames=8, fixed_frames=8) Use meta file to control datasets loading. Data path will use as meta path 08/26/2026 19:33:00 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path 08/26/2026 19:33:00 - INFO - root - Use meta file to control datasets loading. Data path will use as meta path Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loading data from ['anno_online/detail_caption_recap.json'], dataset name: internVid, dataset root: /share/dataset/vision_feat_cache Loading anno_online/detail_caption_recap.json via `load_dataset` Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples Loaded 977344 samples /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. /root/OnlineLong/videollama3/train/videollama3_trainer.py:227: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `VideoLLaMA3Trainer.__init__`. Use `processing_class` instead. super().__init__(*args, **kwargs) The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. wandb: Loading settings from /root/OnlineLong/wandb/settings wandb: Tracking run with wandb version 0.28.2 wandb: W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing. wandb: Run data is saved locally in /root/OnlineLong/wandb/offline-run-20260826_193447-rykxtm3u wandb: View this run in the terminal with `wandb leet` 0%| | 0/7636 [00:00