How to use Composio/mixtral_tensorrt_a100_tp2_w2_paged_kvcache with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Composio/mixtral_tensorrt_a100_tp2_w2_paged_kvcache", device_map="auto")
https://huggingface.co/Composio/mixtral_tensorrt_a100_tp2_w2_paged_kvcache/resolve/main/model.cache
hf download hf://Composio/mixtral_tensorrt_a100_tp2_w2_paged_kvcache/model.cache
curl -L -o model.cache https://huggingface.co/Composio/mixtral_tensorrt_a100_tp2_w2_paged_kvcache/resolve/main/model.cache
