File size: 926 Bytes
2b0945e 522cdd9 2b0945e 017391b 40309c6 c1455ff 40309c6 5319afa 28198cc 5319afa 28198cc 5319afa 28198cc 5319afa 28198cc 5319afa 28198cc 5319afa 28198cc 16ca49a 49670bb 8e44694 49670bb d80b8ee | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | ---
license: mit
language:
- en
base_model:
- microsoft/Phi-3-mini-128k-instruct
pipeline_tag: text-generation
---
## GGUF quantized version of Phi-3 Model (128k-instruct mini)
project original [source](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) (base model)
Q_2 (not nice)
Q_3 (acceptable)
Q_4 family is recommanded (good for running with CPU as well)
Q_5 (good in general)
Q_6 is good also; if you want a better result; take this one instead of Q_5
Q_8 which is very good; need a reasonable size of RAM otherwise you might expect a long wait
16-bit and 32-bit are also provided here for research perspectives; since the file size (16bit) is similar to the original safetensors; once you have a GPU, go ahead with the safetensors, pretty much the same
### how to run it
use any connector for interacting with gguf; i.e., [gguf-connector](https://pypi.org/project/gguf-connector/)
welcome to ai era |