Card: base_model_relation: quantized (so the conversion lists under the base model's Quantizations, not Finetunes)
d4342dd verified | license: mit | |
| tags: | |
| - executorch | |
| - xnnpack | |
| - pte | |
| - on-device | |
| - text-generation | |
| base_model: | |
| - microsoft/Phi-4-mini-instruct | |
| base_model_relation: quantized | |
| # Phi-4-mini-instruct — ExecuTorch XNNPACK 8da4w + 8-bit embedding | |
| `phi_4_mini_xnnpack_8da4w_e8.pte` (2597.9 MB) | |
| - **Source**: Qwen/Phi-4-mini-instruct | |
| - **License**: Apache-2.0 | |
| - **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`) | |
| - **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048, | |
| XNNPACK extended_ops | |
| - **Config**: `llm_params/phi_4_mini_xnnpack_8da4w_e8.yaml` | |
| ## Verification (Mac arm64, 2026-08-21) | |
| `llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per | |
| prompt so no answer is read through the previous one's cache: | |
| | prompt | answer | decode | | |
| |---|---|---| | |
| | capital of France? | "The capital of France is Paris." then keeps going | 49.3 tok/s | | |
| | 17 times 4? | "17 times 4 is equal to 68." and stops | 49.7 tok/s | | |
| Stopping is the part worth looking at. Chat template is `<|user|>...<|end|><|assistant|>`, | |
| bos 199999, eos [200020, 199999] — the model's own | |
| `generation_config.json`. The upstream example config | |
| (`examples/models/phi_4_mini/config/phi_4_mini_xnnpack.yaml`) carries 151643 for both, which | |
| is Qwen's, and a model that never sees its end token does not stop. | |
| Not measured on a phone. Worth noting against the rest of the shelf: this decodes at 49 | |
| tok/s where [Qwen3.5-4B](https://huggingface.co/mlboydaisuke/Qwen3.5-4B-ExecuTorch) manages | |
| 8.6 at a comparable 2.8 GB. The Qwen3.5 exports run with `use_sdpa_with_kv_cache: False` | |
| and this one with it on. | |
| ## Converting it | |
| ```bash | |
| python convert/export_phi_4_mini.py | |
| ``` | |
| Not `export_llm` directly, because its download path cannot reach this model: | |
| `load_checkpoint_from_pytorch_model` reads `pytorch_model.bin` and this repository ships | |
| safetensors only, so the code falls back to torchtune's checkpointer — a dependency with | |
| its own torch pin, for a key rename and two splits. The script reads the safetensors, calls | |
| upstream's own `phi_4_hf_to_meta`, and ties the output embedding back on: input and output | |
| embeddings are tied here, so there is no `lm_head.weight` to map to `output.weight`, and the | |
| model refuses a checkpoint without it. The torchtune converter beside it ties them; the | |
| safetensors one does not. | |
| ## Running it | |
| ```bash | |
| python llm_params/gen_static.py \ | |
| --pte phi_4_mini_xnnpack_8da4w_e8.pte \ | |
| --tokenizer tokenizer.json \ | |
| --prompt '<|user|>What is the capital of France?<|end|><|assistant|>' \ | |
| --eos_ids "[200020,199999]" | |
| ``` | |
| The 8-bit embedding needs `from executorch.kernels import quantized` before the program is | |
| loaded. Without it the method will not even load — `kernel | |
| 'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken | |
| export rather than a runtime missing its kernels. | |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) · | |
| iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples)) | |