Phi-4-mini-instruct-Autoround-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the Phi-4-mini-instruct-Autoround-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

• Model architecture: Phi-4 Mini
• Quantization: A16W4 (BF16 activations, INT4 language weights), with mixed-precision exceptions retained from the prequantized checkpoint.
• Pre-quantization: The source checkpoint was pre-quantized with AutoRound before compilation.
• Maximum context length: 4096 tokens
• Source model: microsoft/Phi-4-mini-instruct

Performance

The following measurements characterize text generation on Modalix.

Measured with MoLE using batch size 1, four samples per input length, and max_new_tokens=128. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.

Input tokens	Mean TTFT (seconds)	Mean generation rate (tokens/second)
128	0.13	21.65
256	0.22	21.43
512	0.44	21.02
1024	0.90	20.23
2048	1.99	17.40
3072	3.72	15.97

Prerequisites

To run this model, you need a SiMa.ai Modalix device with the SiMa.ai Neat Runtime installed.

Installation and Deployment

Download the precompiled model directly on Modalix:

hf download simaai/Phi-4-mini-instruct-Autoround-a16w4 --local-dir /media/nvme/llima/models/Phi-4-mini-instruct-Autoround-a16w4

The model is installed under:

/media/nvme/llima/models/Phi-4-mini-instruct-Autoround-a16w4

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/Phi-4-mini-instruct-Autoround-a16w4 --local-dir Phi-4-mini-instruct-Autoround-a16w4
scp -r Phi-4-mini-instruct-Autoround-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Usage

Validate the model directly on Modalix:

llima run Phi-4-mini-instruct-Autoround-a16w4

For application integration, see GenAI with LLiMa.

Recompilation

To build for an 8,192-token context, recompile the matching source checkpoint with this repository's compile.sh in a compatible SiMa Model Compiler and LLiMa compilation environment. Change --max_num_tokens 4096 to --max_num_tokens 8192 in the script, then follow its source-checkpoint and output-directory instructions.

The context budget includes input and generated tokens. A larger context increases KV-cache memory requirements and must fit the target device. The downloadable artifacts use a 4,096-token context; any performance measurements in this card refer to that build.

Limitations

• This repository contains compiled Modalix artifacts and cannot be loaded with transformers.AutoModel.
• Quantization can cause minor differences from the full-precision source model.

Resources

• GenAI with LLiMa
• Serve GenAI Models
• Run an LLM
• Source model