runtime error

Exit code: 1. Reason: ges/transformers/modeling_utils.py", line 4368, in from_pretrained loading_info, disk_offload_index = cls._load_pretrained_model(model, state_dict, checkpoint_files, load_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4502, in _load_pretrained_model loading_info, disk_offload_index = convert_and_load_state_dict_in_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1660, in convert_and_load_state_dict_in_model realized_value = mapping.convert( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1130, in convert collected_tensors = self.materialize_tensors() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/transformers/core_model_loading.py", line 925, in materialize_tensors tensors = [func() for func in tensors] ^^^^^^ File "/usr/local/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1199, in _job return _materialize_copy(tensor, device, dtype) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1185, in _materialize_copy tensor = tensor.to(device=device, dtype=dtype) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDACachingAllocator.cpp":1131, please report a bug to PyTorch. Traceback (most recent call last): File "/home/user/app/app.py", line 23, in <module> _load_model() File "/usr/local/lib/python3.12/site-packages/spaces/zero/wrappers.py", line 238, in gradio_handler raise error("ZeroGPU worker error", res.error_cls) gradio.exceptions.Error: 'RuntimeError'

Container logs: