runtime error

Exit code: 1. Reason: se logging (usually --v=1 or --vmodule=cuda_diagnostics=1) to get more diagnostic output from this module I0000 00:00:1784611218.139867 1 cuda_diagnostics.cc:176] retrieving CUDA diagnostic information for host: r-realmanise-garbage-classifier-23zreoqo-bd836-snsxz I0000 00:00:1784611218.139871 1 cuda_diagnostics.cc:183] hostname: r-realmanise-garbage-classifier-23zreoqo-bd836-snsxz I0000 00:00:1784611218.139936 1 cuda_diagnostics.cc:190] libcuda reported version is: 580.126.9 I0000 00:00:1784611218.139947 1 cuda_diagnostics.cc:194] kernel reported version is: 580.126.9 I0000 00:00:1784611218.139949 1 cuda_diagnostics.cc:284] kernel version seems to match DSO: 580.126.9 /usr/local/lib/python3.10/site-packages/keras/src/saving/saving_lib.py:843: UserWarning: Skipping variable loading for optimizer 'rmsprop', because it has 8 variables whereas the saved optimizer has 14 variables. saveable.load_own_variables(weights_store.get(inner_path)) SPACES_ZERO_GPU_DEBUG self.arg_queue._writer.fileno()=10 SPACES_ZERO_GPU_DEBUG self.res_queue._writer.fileno()=12 Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 148, in worker_init torch.init(nvidia_uuid) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 417, in init torch.Tensor([0]).cuda() File "/usr/local/lib/python3.10/site-packages/torch/cuda/__init__.py", line 478, in _lazy_init torch._C._cuda_init() RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 302: Error loading CUDA libraries. GPU will not be used. Traceback (most recent call last): File "/home/user/app/app.py", line 64, in <module> initialize_gpu_hardware() File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 238, in gradio_handler raise error("ZeroGPU worker error", res.error_cls) gradio.exceptions.Error: 'RuntimeError'

Container logs:

Fetching error logs...