Quantizing https://huggingface.co/inclusionAI/Ling-3.0-flash-Sante
#3022
by Grujowmi - opened
Hi! Would you consider quantizing https://huggingface.co/inclusionAI/Ling-3.0-flash-Sante for use with vLLM?
Hardware: DGX SPARK
vLLM version: dont care :)
Preferred format: INT4 / FP8
Ideally, could you include a tested vllm serve command? Thanks!
hey, we are doing only gguf. we queued this model a few days ago but it failed. so we will wait a couple days and queue it again. We are not providing awq or other quants, sorry. feel free to request more gguf models tho = )