Irfanuruchi/Llama-3.2-1B-Computer-Engineering-OpenVINO-INT8

Llama-3.2-1B Computer Engineering - OpenVINO INT8

OpenVINO INT8 conversion of Irfanuruchi/Llama-3.2-1B-Computer-Engineering-LLM, a 1B-parameter model specialized for Computer Engineering.

The model was converted and tested on IULinux with OpenVINO 2026.4 across three Intel execution backends:

• Intel Core Ultra 9 275HX CPU
• Intel Graphics integrated GPU
• Intel AI Boost NPU

The resulting OpenVINO model directory is approximately 1.2 GB.

Prompt format

This model works best as a completion-style Q/A model rather than as a general-purpose chat assistant.

Recommended format:

Q: <computer engineering question>
A:

Example:

Q: Explain the purpose of CPU cache memory.
A:

Recommended generation settings:

max_new_tokens = 200
temperature = 0.7
top_p = 0.9
do_sample = True
repetition_penalty = 1.1

Short, direct technical questions generally work better than role-based conversational prompts.

Usage

Install OpenVINO GenAI:

python -m pip install "openvino==2026.4" "openvino-genai==2026.4"

Example:

import openvino_genai as ov_genai

model_path = "./Llama-3.2-1B-Computer-Engineering-OpenVINO-INT8"

device = "CPU"

pipe = ov_genai.LLMPipeline(
    model_path,
    device,
)

config = ov_genai.GenerationConfig()

config.max_new_tokens = 200
config.do_sample = True
config.temperature = 0.7
config.top_p = 0.9
config.repetition_penalty = 1.1
config.apply_chat_template = False

question = "Explain the purpose of CPU cache memory."

prompt = f"""Q: {question}
A:"""

result = pipe.generate(
    prompt,
    config,
)

print(result)

Available OpenVINO devices can be checked with:

import openvino as ov

core = ov.Core()
print(core.available_devices)

On the IULinux system used for this release:

['CPU', 'GPU', 'NPU']

Conversion

The published parent model is a genuine BitsAndBytes INT8 checkpoint.

Inspection of the parent model showed INT8 linear weights, SCB scaling tensors, and 112 Linear8bitLt modules.

The conversion path was:

meta-llama/Llama-3.2-1B
        ↓
Computer Engineering fine-tuning
        ↓
BitsAndBytes INT8 parent checkpoint
        ↓
Dequantized dense FP16 representation
        ↓
OpenVINO INT8

After dequantization, the intermediate checkpoint contained:

146 FP16 tensors
0 BitsAndBytes linear modules
0 INT8 weight tensors
0 SCB tensors

The OpenVINO export used:

optimum-cli export openvino \
  --model ./source-dequant-fp16 \
  --task text-generation-with-past \
  --weight-format int8 \
  --sym \
  ./Llama-3.2-1B-Computer-Engineering-OpenVINO-INT8

The OpenVINO export used symmetric per-channel INT8 weight compression.

Because the parent model was already quantized, this model should not be described as a direct conversion from the original pre-quantization FP16 weights.

The conversion path includes a lossy BitsAndBytes INT8 → dense FP16 reconstruction before OpenVINO INT8 compression.

Size

Observed local sizes during conversion:

Variant	Approximate size
OpenVINO FP16	2.4 GB
OpenVINO INT8	1.2 GB
OpenVINO INT4	758 MB

Benchmarks

Benchmarks were run on IULinux using the same prompt and generation workload across CPU, integrated GPU, and NPU.

System

Component	Configuration
OS	IULinux 0.1 Development
Kernel	7.0.0-31-generic
CPU	Intel Core Ultra 9 275HX
GPU	Intel Graphics (iGPU)
NPU	Intel AI Boost
OpenVINO	2026.4
OpenVINO GenAI	2026.4
Power profile	Performance
Power source	AC

OpenVINO exposed:

CPU: Intel(R) Core(TM) Ultra 9 275HX
GPU: Intel(R) Graphics (iGPU)
NPU: Intel(R) AI Boost

Method

Prompt tokens: 26
Generated tokens: 128
Warm-up runs: 2
Measured runs: 5
do_sample: false
ignore_eos: true

Benchmark prompt:

Q: Explain how CPU cache memory improves processor performance, including its relationship with RAM, latency, and memory locality.
A:

ignore_eos=True was used so every backend performed the same 128-token generation workload.

Greedy decoding was used only for deterministic performance measurement. The recommended sampling settings above should be used for normal model interaction.

Results

Device	Pipeline load	TTFT	TPOT	Throughput
Intel Core Ultra 9 275HX	321.26 ms	44.24 ± 4.97 ms	19.75 ± 0.06 ms/token	50.64 ± 0.16 tok/s
Intel Graphics iGPU	1859.25 ms	34.60 ± 0.12 ms	23.89 ± 0.06 ms/token	41.86 ± 0.11 tok/s
Intel AI Boost NPU	5653.16 ms	922.20 ± 1.20 ms	29.15 ± 0.09 ms/token	34.30 ± 0.11 tok/s

On this INT8 workload, the CPU produced the highest generation throughput.

The integrated GPU produced the lowest measured TTFT.

The NPU produced very consistent throughput across the five measured runs.

No device-level power measurements were collected, so these results compare latency and throughput only.

Precision comparison

All three OpenVINO variants were tested using the same benchmark methodology.

Precision	CPU	Intel iGPU	Intel NPU
INT4	62.98 tok/s	62.59 tok/s	33.89 tok/s
INT8	50.64 tok/s	41.86 tok/s	34.30 tok/s
FP16	26.34 tok/s	24.43 tok/s	20.63 tok/s

INT4 produced the highest measured CPU and integrated-GPU throughput.

INT8 produced the highest measured NPU throughput, although the difference between INT8 and INT4 on the NPU was small.

IULinux

The complete conversion and validation workflow was performed on IULinux.

The same installation exposed all three Intel OpenVINO execution backends:

CPU
GPU
NPU

The Intel AI Boost NPU used the Intel NPU Linux driver stack and Level Zero runtime.

The integrated GPU used the existing i915 kernel driver together with the Intel OpenCL compute runtime.

No NVIDIA driver modification was required for this OpenVINO deployment.

Quality checks

The model behaves substantially better using its intended completion-style Q/A format:

Q: <question>
A:

The OpenVINO INT8 variant produced an average repeated 4-gram ratio of:

0.000

For comparison:

Variant	Average repeated 4-gram ratio
OpenVINO FP16	0.002
OpenVINO INT8	0.000
OpenVINO INT4	0.003

These are diagnostic repetition measurements and should not be interpreted as formal model-quality scores.

Files

The repository contains the OpenVINO model, tokenizer and detokenizer IR files, configuration files, benchmark data, environment information, and SHA-256 checksums.

Benchmark and reproducibility files are stored under:

benchmarks/

Limitations

This is a specialized Computer Engineering completion model and is not intended to behave like a general-purpose conversational assistant.

Technical answers can contain factual errors and should be independently verified for critical engineering work.

The model may continue generating additional Q/A pairs after completing an answer. Reducing max_new_tokens can help control output length.

The OpenVINO INT8 model was derived from a previously quantized BitsAndBytes INT8 checkpoint through an intermediate dense FP16 reconstruction.

Quantization and dequantization are lossy operations.

Benchmark results apply only to the tested hardware and software configuration.

License

This model is governed by the Llama 3.2 Community License.

Use, modification, and redistribution must comply with the Llama 3.2 Community License and its associated requirements.

Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Parent model

Irfanuruchi/Llama-3.2-1B-Computer-Engineering-LLM

Citation

@misc{uruchi2026llama32ceopenvinoint8,
  title     = {Llama-3.2-1B-Computer-Engineering-OpenVINO-INT8},
  author    = {Irfanuruchi},
  year      = {2026},
  publisher = {Hugging Face}
}