|
Download README.md from Nekodeus/phi35-mini-instruct-openvino: direct link, hf CLI and curl.
- Browser
- Download file 1.46 kB
-
https://huggingface.co/Nekodeus/phi35-mini-instruct-openvino/resolve/main/README.md
- Command line
-
hf download hf://Nekodeus/phi35-mini-instruct-openvino/README.md
-
curl -L -o README.md https://huggingface.co/Nekodeus/phi35-mini-instruct-openvino/resolve/main/README.md
1.46 kB
| license: mit | |
| library_name: openvino | |
| tags: | |
| - openvino | |
| - int8 | |
| - text-generation | |
| - phi-3 | |
| - instruct | |
| - llm | |
| # Phi-3.5-mini-instruct OpenVINO (INT8) | |
| OpenVINO IR conversion of [microsoft/Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) with weight-only INT8 quantization, produced on CPU-only Kaggle with `optimum-cli` (fully offline, token-free). | |
| - Base model: `microsoft/Phi-3.5-mini-instruct` (Phi3ForCausalLM, 3.8B params, MIT) | |
| - Task: `text-generation` | |
| - Precision: weights INT8 (NNCF), activations FP16/FP32 as traced | |
| - Converted with: `optimum-intel 2.2.0`, `openvino 2026.4.1`, `transformers 5.16.1` | |
| - Context: 128k tokens (longrope scaling) | |
| ## Usage | |
| ### optimum-intel | |
| ```python | |
| from optimum.intel import OVModelForCausalLM | |
| from transformers import AutoTokenizer | |
| model = OVModelForCausalLM.from_pretrained("Nekodeus/phi35-mini-instruct-openvino") | |
| tok = AutoTokenizer.from_pretrained("Nekodeus/phi35-mini-instruct-openvino") | |
| inputs = tok("<|user|>\nExplain recursion in programming.<|end|>\n<|assistant|>\n", return_tensors="pt") | |
| print(tok.decode(model.generate(**inputs, max_new_tokens=256)[0])) | |
| ``` | |
| ### OpenVINO GenAI | |
| ```cpp | |
| ov::genai::LLMPipeline pipe("path/to/Nekodeus/phi35-mini-instruct-openvino", "CPU"); | |
| ``` | |
| ## Reproduce | |
| ```bash | |
| optimum-cli export openvino \ | |
| --model microsoft/Phi-3.5-mini-instruct \ | |
| --task text-generation \ | |
| --weight-format int8 \ | |
| phi35-mini-instruct-openvino | |
| ``` | |