Qwen/Qwen3.8-27B | INT4 (W4A16)
#55
by INC4AI - opened
Pipeline Failure Report
Model: Qwen/Qwen3.8-27B
Quantization Scheme: INT4 (W4A16)
Failed Phase: quantize
Run ID: Qwen3.8-27B-AutoRound-W4A16-Tuning
Error Category: out_of_memory
Full Error Log
10:36:47 [INFO] HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3.8-27B/xet-read-token/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 "HTTP/1.1 200 OK"
10:36:48 [INFO] HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/tokenizer.model "HTTP/1.1 404 Not Found"
10:36:48 [INFO] HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/added_tokens.json "HTTP/1.1 404 Not Found"
10:36:48 [INFO] HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/special_tokens_map.json "HTTP/1.1 404 Not Found"
10:36:49 [INFO] HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja "HTTP/1.1 307 Temporary Redirect"
10:36:49 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.8-27B/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja?%2FQwen%2FQwen3.8-27B%2Fresolve%2F1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0%2Fchat_template.jinja=&etag=%22c0c686f9c38d70d179fb7b5f5aa7530bc913dda3%22 "HTTP/1.1 200 OK"
10:36:49 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.8-27B/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja?%2FQwen%2FQwen3.8-27B%2Fresolve%2F1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0%2Fchat_template.jinja=&etag=%22c0c686f9c38d70d179fb7b5f5aa7530bc913dda3%22 "HTTP/1.1 200 OK"
10:36:49 [INFO] HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3.8-27B "HTTP/1.1 200 OK"
10:36:49 [INFO] Loading model...
10:36:49 [INFO] HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3.8-27B "HTTP/1.1 200 OK"
10:36:50 [INFO] HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/model.safetensors "HTTP/1.1 404 Not Found"
10:36:50 [INFO] HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/model.safetensors.index.json "HTTP/1.1 307 Temporary Redirect"
10:36:50 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.8-27B/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/model.safetensors.index.json?%2FQwen%2FQwen3.8-27B%2Fresolve%2F1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0%2Fmodel.safetensors.index.json=&etag=%22da35e3c564457dface7d138f0b6cac284ff8958c%22 "HTTP/1.1 200 OK"
10:36:50 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.8-27B/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/model.safetensors.index.json?%2FQwen%2FQwen3.8-27B%2Fresolve%2F1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0%2Fmodel.safetensors.index.json=&etag=%22da35e3c564457dface7d138f0b6cac284ff8958c%22 "HTTP/1.1 200 OK"
10:36:50 [INFO] HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3.8-27B/tree/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0?recursive=true&expand=false "HTTP/1.1 200 OK"
[W1011 10:37:26.504056205 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 178257920 bytes (free: 62783488, total: 33668857856).
[W1011 10:37:26.586795649 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 178257920 bytes (free: 62783488, total: 33668857856).
[W1011 10:37:26.602740104 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 104857600 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.602787112 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 104857600 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.602909621 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 62914560 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.602946320 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 62914560 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.603101430 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 62914560 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.603140783 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 62914560 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.611698766 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 178257920 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.611729774 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 178257920 bytes (free: 60686336, total: 33668857856).
10:37:26 [ERROR] Quantization failed: CUDA out of memory. Tried to allocate 170.00 MiB. GPU 0 has a total capacity of 31.36 GiB of which 59.88 MiB is free. Including non-PyTorch memory, this process has 31.29 GiB memory in use. Of the allocated memory 30.80 GiB is allocated by PyTorch, and 2.80 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
[W1011 10:37:26.613114641 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 178257920 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.613146010 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 178257920 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.613236870 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 178257920 bytes (free: 60686336, total: 33668857856).
[W1011 10:37:26.613266305 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 178257920 bytes (free: 60686336, total: 33668857856).
Traceback (most recent call last):
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 479, in <module>
quantize(args)
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 293, in quantize
model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/auto_round/utils/common.py", line 205, in patched
return underlying_func(klass, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 411, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4338, in from_pretrained
loading_info, disk_offload_index = cls._load_pretrained_model(model, state_dict, checkpoint_files, load_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4481, in _load_pretrained_model
loading_info, disk_offload_index = convert_and_load_state_dict_in_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1754, in convert_and_load_state_dict_in_model
realized_value = mapping.convert(
^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1008, in convert
collected_tensors = self.materialize_tensors()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 970, in materialize_tensors
tensors = [future.result() for future in tensors if future.result() is not None]
^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1262, in _job
return _materialize_copy(tensor, device, dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1240, in _materialize_copy
tensor = tensor.to(device=device, dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 170.00 MiB. GPU 0 has a total capacity of 31.36 GiB of which 59.88 MiB is free. Including non-PyTorch memory, this process has 31.29 GiB memory in use. Of the allocated memory 30.80 GiB is allocated by PyTorch, and 2.80 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Auto-generated by error_analysis pipeline. cc @lvkaokao
Looks like Qwen/Qwen3.8-27B | INT4 (W4A16) hit a CUDA out of memory error on GPU 0, which has only ~59.88 MiB free despite a 31.36 GiB capacity. The traceback shows the failure during tensor device transfer in core_model_loading.py. If you want to proceed with this model or need help adjusting the batch size, quantization settings, or memory allocation flags like PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, let me know—I can draft a concise fix or summarize the error context for your team.