add README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: openvino
|
| 4 |
+
tags:
|
| 5 |
+
- openvino
|
| 6 |
+
- int8
|
| 7 |
+
- text-generation
|
| 8 |
+
- phi-3
|
| 9 |
+
- instruct
|
| 10 |
+
- llm
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# Phi-3.5-mini-instruct OpenVINO (INT8)
|
| 14 |
+
|
| 15 |
+
OpenVINO IR conversion of [microsoft/Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) with weight-only INT8 quantization, produced on CPU-only Kaggle with `optimum-cli` (fully offline, token-free).
|
| 16 |
+
|
| 17 |
+
- Base model: `microsoft/Phi-3.5-mini-instruct` (Phi3ForCausalLM, 3.8B params, MIT)
|
| 18 |
+
- Task: `text-generation`
|
| 19 |
+
- Precision: weights INT8 (NNCF), activations FP16/FP32 as traced
|
| 20 |
+
- Converted with: `optimum-intel 2.2.0`, `openvino 2026.4.1`, `transformers 5.16.1`
|
| 21 |
+
- Context: 128k tokens (longrope scaling)
|
| 22 |
+
|
| 23 |
+
## Usage
|
| 24 |
+
|
| 25 |
+
### optimum-intel
|
| 26 |
+
|
| 27 |
+
```python
|
| 28 |
+
from optimum.intel import OVModelForCausalLM
|
| 29 |
+
from transformers import AutoTokenizer
|
| 30 |
+
|
| 31 |
+
model = OVModelForCausalLM.from_pretrained("Nekodeus/phi35-mini-instruct-openvino")
|
| 32 |
+
tok = AutoTokenizer.from_pretrained("Nekodeus/phi35-mini-instruct-openvino")
|
| 33 |
+
inputs = tok("<|user|>\nExplain recursion in programming.<|end|>\n<|assistant|>\n", return_tensors="pt")
|
| 34 |
+
print(tok.decode(model.generate(**inputs, max_new_tokens=256)[0]))
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
### OpenVINO GenAI
|
| 38 |
+
|
| 39 |
+
```cpp
|
| 40 |
+
ov::genai::LLMPipeline pipe("path/to/Nekodeus/phi35-mini-instruct-openvino", "CPU");
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
## Reproduce
|
| 44 |
+
|
| 45 |
+
```bash
|
| 46 |
+
optimum-cli export openvino \
|
| 47 |
+
--model microsoft/Phi-3.5-mini-instruct \
|
| 48 |
+
--task text-generation \
|
| 49 |
+
--weight-format int8 \
|
| 50 |
+
phi35-mini-instruct-openvino
|
| 51 |
+
```
|