REQUEST!: FP8 Quant of this model to run in SGLang was needed with MTP and vision head attached.

#1
by ramachandrajoshi - opened

I have tested the model gguf mxfp4 version and liked the responses.
So i want to check the FP8 quant quality. Anyone can quantize this model to FP8 compatible with SGLang or VLLM?

hi, the mtp head was not post-trained together with the model itself. I will fine tune it next week and get back to you then.

Sign up or log in to comment