- lazy bandaid to fix Mistral's gated config.json being inaccessible (c9ce7ade6670ccbd2881dbdd64002b4255395b10)
Co-authored-by: Matt <mo137@users.noreply.huggingface.co>
Add support for EXL2 4 bit KV cache; switch from metric gigabytes (1e9 bytes) to JEDEC gigabytes (2^30 bytes) (#2)
- Add support for EXL2 4 bit KV cache; switch from metric gigabytes (1e9 bytes) to JEDEC gigabytes (2^30 bytes) (c8c7129688460a9f9ecab851edacde8933618778)
Co-authored-by: Matt <mo137@users.noreply.huggingface.co>