Nekodeus commited on
Commit
3b2ff01
·
verified ·
1 Parent(s): 72d615e

add README.md

Browse files
Files changed (1) hide show
  1. README.md +51 -0
README.md ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ library_name: openvino
4
+ tags:
5
+ - openvino
6
+ - int8
7
+ - text-generation
8
+ - phi-3
9
+ - instruct
10
+ - llm
11
+ ---
12
+
13
+ # Phi-3.5-mini-instruct OpenVINO (INT8)
14
+
15
+ OpenVINO IR conversion of [microsoft/Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) with weight-only INT8 quantization, produced on CPU-only Kaggle with `optimum-cli` (fully offline, token-free).
16
+
17
+ - Base model: `microsoft/Phi-3.5-mini-instruct` (Phi3ForCausalLM, 3.8B params, MIT)
18
+ - Task: `text-generation`
19
+ - Precision: weights INT8 (NNCF), activations FP16/FP32 as traced
20
+ - Converted with: `optimum-intel 2.2.0`, `openvino 2026.4.1`, `transformers 5.16.1`
21
+ - Context: 128k tokens (longrope scaling)
22
+
23
+ ## Usage
24
+
25
+ ### optimum-intel
26
+
27
+ ```python
28
+ from optimum.intel import OVModelForCausalLM
29
+ from transformers import AutoTokenizer
30
+
31
+ model = OVModelForCausalLM.from_pretrained("Nekodeus/phi35-mini-instruct-openvino")
32
+ tok = AutoTokenizer.from_pretrained("Nekodeus/phi35-mini-instruct-openvino")
33
+ inputs = tok("<|user|>\nExplain recursion in programming.<|end|>\n<|assistant|>\n", return_tensors="pt")
34
+ print(tok.decode(model.generate(**inputs, max_new_tokens=256)[0]))
35
+ ```
36
+
37
+ ### OpenVINO GenAI
38
+
39
+ ```cpp
40
+ ov::genai::LLMPipeline pipe("path/to/Nekodeus/phi35-mini-instruct-openvino", "CPU");
41
+ ```
42
+
43
+ ## Reproduce
44
+
45
+ ```bash
46
+ optimum-cli export openvino \
47
+ --model microsoft/Phi-3.5-mini-instruct \
48
+ --task text-generation \
49
+ --weight-format int8 \
50
+ phi35-mini-instruct-openvino
51
+ ```