"Self-speculative decoding" not supported?

#1
by PureMattness - opened

I noticed your quants explicitly say they don't have speculative decoding - does this mean these quants do not include the "self-speculative decoding" mentioned in Diffuspace's original writeup?

Hmm good question, there isn't explicit MTP support, their model card points to this repo for the self speculative decoding:

https://github.com/DiffuSpace/Clover-35B-A3B-Preview

And in it they say llama.cpp is supported but also that "Backend code and setup guides will be added to this repository", so maybe they'll need to push new code?

working with llama to get a working version of self spec decoding

Sign up or log in to comment