unable to save AOT compiled function

by mohamedemam - opened Jun 5

Jun 5

unable to save AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/58149929561eb15272d8ce6245f1e07e5d9a413fd0c51d06ed7555b8fed32ff4/rank_0_0/model: Can't pickle <function Gemma4UnifiedTextModel.forward at 0x712b3f926520>: it's not the same object as transformers.models.gemma4_unified.modeling_gemma4_unified.Gemma4UnifiedTextModel.forward
[core.py:1165] RuntimeError: Shape mismatch: a.size(1) = 4096, size_k = 8192

pannaga10

Google org Jun 9

Hi @mohamedemam
Thanks for reporting this. Are you by any chance using vLLM v0.22.0 version ? If yes there is a similar bug filed here . https://github.com/vllm-project/vllm/issues/44796.
Can you check this once . Thanks

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment