Recommendation for 8gb vram?

#5
by idkgoogleit - opened

Amazing, can't wait to use it in llama.cpp

Any recommendations for 8gb vram perhaps, or is 12gb vram due to looping really needed?

idkgoogleit changed discussion title from Recommendation for ?8gb vram? to Recommendation for 8gb vram?

Thanks for your interest! We expect to release the quantized versions of the model along with the corresponding llama.cpp support tomorrow.

We're also exploring KV cache rewriting strategies for the looped architecture to further reduce inference memory requirements, with the goal of bringing VRAM usage below 8 GB. We'll share more updates soon!

Sign up or log in to comment