Intel hardware compatible?
Hello, does it run on any Intel Hardware like Intel Arc iGPU and GPU?
Thanks
I tried it on 4xB60 cards and is giving issues like KeyError: 'layers.0.mlp.experts.routed_experts.w2_qweight' . I got memory error while trying to load it on 2 cards.
please refer this pr currently @AI-Joe-git @Azhuvath https://github.com/vllm-project/vllm/pull/47124
@lvkaokao Do you know when it will be merged to the main branch? Will it work on a single B60 Card (24 GB VRAM)?
I'm also interested if it will work on 24GB of VRAM
The model can technically run on a 24GB B60. However, model loading alone consumes about 19.66 GiB of GPU memory, leaving only a limited amount of memory available for the KV cache. As a result, short-context workloads should be fine, but tasks requiring longer context lengths may not fit and could run into KV cache memory limitations. @Azhuvath @Tretrrr
The model can technically run on a 24GB B60. However, model loading alone consumes about 19.66 GiB of GPU memory, leaving only a limited amount of memory available for the KV cache. As a result, short-context workloads should be fine, but tasks requiring longer context lengths may not fit and could run into KV cache memory limitations. @Azhuvath @Tretrrr
What about on intel ARC IGPU ? i can use 27.4GB of shared "VRAM" ?