Irfanuruchi/Llama-3.2-1B-Computer-Engineering-OpenVINO-INT4

Llama-3.2-1B Computer Engineering - OpenVINO INT4

OpenVINO INT4 conversion of Irfanuruchi/Llama-3.2-1B-Computer-Engineering-LLM, a 1B-parameter model specialized for Computer Engineering.

The model was converted and tested on IULinux with OpenVINO 2026.4 across three Intel execution backends:

• Intel Core Ultra 9 275HX CPU
• Intel Graphics integrated GPU
• Intel AI Boost NPU

The resulting OpenVINO model directory is approximately 758 MB.

Prompt format

This model works best as a completion-style Q/A model rather than as a general-purpose chat assistant.

Recommended format:

Q: <computer engineering question>
A:

Example:

Q: Explain the purpose of CPU cache memory.
A:

Recommended generation settings:

max_new_tokens = 200
temperature = 0.7
top_p = 0.9
do_sample = True
repetition_penalty = 1.1

Short, direct technical questions generally work better than role-based conversational prompts.

Usage

Install OpenVINO GenAI:

python -m pip install "openvino==2026.4" "openvino-genai==2026.4"

Example:

import openvino_genai as ov_genai

model_path = "./Llama-3.2-1B-Computer-Engineering-OpenVINO-INT4"

# CPU, GPU, or NPU depending on available OpenVINO devices.
device = "CPU"

pipe = ov_genai.LLMPipeline(
    model_path,
    device,
)

config = ov_genai.GenerationConfig()

config.max_new_tokens = 200
config.do_sample = True
config.temperature = 0.7
config.top_p = 0.9
config.repetition_penalty = 1.1
config.apply_chat_template = False

question = "Explain the purpose of CPU cache memory."

prompt = f"""Q: {question}
A:"""

result = pipe.generate(
    prompt,
    config,
)

print(result)

Available devices can be checked with:

import openvino as ov

core = ov.Core()
print(core.available_devices)

On the IULinux test system used for this release:

['CPU', 'GPU', 'NPU']

Conversion

The parent model is a genuine BitsAndBytes INT8 checkpoint rather than a floating-point checkpoint carrying only quantization metadata.

Inspection of the parent model showed INT8 linear weights, SCB scaling tensors, and 112 Linear8bitLt modules.

The conversion path was:

meta-llama/Llama-3.2-1B
        ↓
Computer Engineering fine-tuning
        ↓
BitsAndBytes INT8 parent checkpoint
        ↓
Dequantized dense FP16 representation
        ↓
OpenVINO INT4 weight compression

After dequantization, the intermediate checkpoint contained:

146 FP16 tensors
0 BitsAndBytes linear modules
0 INT8 weight tensors
0 SCB tensors

The OpenVINO conversion used:

optimum-cli export openvino \
  --model ./source-dequant-fp16 \
  --task text-generation-with-past \
  --weight-format int4 \
  --sym \
  --ratio 1.0 \
  --group-size 128 \
  ./Llama-3.2-1B-Computer-Engineering-OpenVINO-INT4

NNCF reported that all 112 ratio-defining layers were compressed using:

INT4 symmetric
group size 128
ratio 1.0

A non-ratio-defining component used INT8 asymmetric backup precision.

Because the parent model was already quantized, this model should not be described as a direct conversion from the original pre-quantization FP16 weights. The conversion includes a lossy INT8 → dense FP16 reconstruction before INT4 compression.

Size

Observed local sizes during conversion:

Variant	Approximate size
Dense FP16	2.4 GB
OpenVINO FP16	2.4 GB
OpenVINO INT8	1.2 GB
OpenVINO INT4	758 MB

Benchmarks

Benchmarks were run on IULinux using the same model, prompt, and generation workload on all three devices.

System

Component	Configuration
OS	IULinux 0.1 Development
Kernel	7.0.0-31-generic
CPU	Intel Core Ultra 9 275HX
GPU	Intel Graphics (iGPU)
NPU	Intel AI Boost
OpenVINO	2026.4
OpenVINO GenAI	2026.4
Power profile	Performance
Power source	AC

OpenVINO exposed:

CPU: Intel(R) Core(TM) Ultra 9 275HX
GPU: Intel(R) Graphics (iGPU)
NPU: Intel(R) AI Boost

Method

Each backend used:

Prompt tokens: 26
Generated tokens: 128
Warm-up runs: 2
Measured runs: 5
do_sample: false
ignore_eos: true

Benchmark prompt:

Q: Explain how CPU cache memory improves processor performance, including its relationship with RAM, latency, and memory locality.
A:

ignore_eos=True was used so every backend performed the same 128-token generation workload.

Greedy decoding was used only for deterministic performance measurement. The recommended sampling configuration above should be used for normal model interaction.

Results

Device	Pipeline load	TTFT	TPOT	Throughput
Intel Core Ultra 9 275HX	382.59 ms	51.57 ± 9.91 ms	15.90 ± 0.74 ms/token	62.98 ± 2.79 tok/s
Intel Graphics iGPU	1218.85 ms	54.24 ± 1.77 ms	15.98 ± 0.29 ms/token	62.59 ± 1.14 tok/s
Intel AI Boost NPU	986.20 ms	1147.08 ± 4.16 ms	29.51 ± 0.10 ms/token	33.89 ± 0.11 tok/s

On this specific 1B INT4 workload, the CPU and integrated GPU produced very similar generation throughput.

The NPU produced lower throughput and substantially higher TTFT in this configuration, but its measured runs were highly consistent.

No device-level power measurements were collected, so these results compare latency and throughput only and should not be interpreted as a power-efficiency comparison.

Results will vary with hardware, firmware, runtime versions, thermal conditions, power configuration, prompt length, and generation settings.

IULinux

The complete conversion and validation workflow was performed on IULinux.

The same IULinux installation exposed all three Intel OpenVINO backends:

CPU
GPU
NPU

The Intel AI Boost NPU used the intel_vpu kernel driver together with the Intel NPU Level Zero userspace stack.

The integrated GPU used the existing i915 kernel driver together with the Intel OpenCL compute runtime.

No NVIDIA driver modifications were required for this OpenVINO deployment.

The benchmark files and environment capture included in this repository record the runtime configuration used for the published results.

Quality checks

The parent model was initially tested using generic greedy prompts and showed significant repetition.

Testing against the original parent checkpoint showed that the model behaves substantially better using its intended Q/A completion format:

Q: <question>
A:

The same prompt format was then tested on the OpenVINO variants.

Observed average repeated 4-gram ratios:

Variant	Average repeated 4-gram ratio
OpenVINO FP16	0.002
OpenVINO INT8	0.000
OpenVINO INT4	0.003

These repetition measurements are diagnostic checks rather than formal model-quality scores.

Files

The repository contains the OpenVINO model, tokenizer and detokenizer IR files, configuration files, benchmark data, environment information, and SHA-256 checksums.

Benchmark files are stored under:

benchmarks/

The final benchmark contains both raw console output and machine-readable JSON.

Limitations

This is a specialized Computer Engineering completion model and is not intended to behave like a general-purpose conversational assistant.

Technical answers can contain factual errors and should be independently verified for critical engineering work.

The model may continue generating additional Q/A pairs after completing an answer. Reducing max_new_tokens can help control output length.

The INT4 model was derived from a previously quantized BitsAndBytes INT8 checkpoint. Quantization and dequantization are lossy operations, and the original pre-quantization precision cannot be reconstructed exactly.

The benchmark results apply only to the tested hardware and software configuration.

License

This model is governed by the Llama 3.2 Community License.

Use, modification, and redistribution must comply with the Llama 3.2 Community License and its associated requirements.

Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Parent model

Irfanuruchi/Llama-3.2-1B-Computer-Engineering-LLM

Citation

@misc{uruchi2026llama32ceopenvino,
  title     = {Llama-3.2-1B-Computer-Engineering-OpenVINO-INT4},
  author    = {Irfanuruchi},
  year      = {2026},
  publisher = {Hugging Face}
}