Nekodeus's picture
add README.md
3b2ff01 verified
|
Raw History Blame Contribute Delete
1.46 kB
---
license: mit
library_name: openvino
tags:
- openvino
- int8
- text-generation
- phi-3
- instruct
- llm
---
# Phi-3.5-mini-instruct OpenVINO (INT8)
OpenVINO IR conversion of [microsoft/Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) with weight-only INT8 quantization, produced on CPU-only Kaggle with `optimum-cli` (fully offline, token-free).
- Base model: `microsoft/Phi-3.5-mini-instruct` (Phi3ForCausalLM, 3.8B params, MIT)
- Task: `text-generation`
- Precision: weights INT8 (NNCF), activations FP16/FP32 as traced
- Converted with: `optimum-intel 2.2.0`, `openvino 2026.4.1`, `transformers 5.16.1`
- Context: 128k tokens (longrope scaling)
## Usage
### optimum-intel
```python
from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer
model = OVModelForCausalLM.from_pretrained("Nekodeus/phi35-mini-instruct-openvino")
tok = AutoTokenizer.from_pretrained("Nekodeus/phi35-mini-instruct-openvino")
inputs = tok("<|user|>\nExplain recursion in programming.<|end|>\n<|assistant|>\n", return_tensors="pt")
print(tok.decode(model.generate(**inputs, max_new_tokens=256)[0]))
```
### OpenVINO GenAI
```cpp
ov::genai::LLMPipeline pipe("path/to/Nekodeus/phi35-mini-instruct-openvino", "CPU");
```
## Reproduce
```bash
optimum-cli export openvino \
--model microsoft/Phi-3.5-mini-instruct \
--task text-generation \
--weight-format int8 \
phi35-mini-instruct-openvino
```