Card: base_model_relation: quantized (so the conversion lists under the base model's Quantizations, not Finetunes)
17ec780 verified | license: other | |
| tags: | |
| - executorch | |
| - xnnpack | |
| - pte | |
| - on-device | |
| - text-generation | |
| base_model: | |
| - LiquidAI/LFM2-700M | |
| base_model_relation: quantized | |
| # LFM2-700M — ExecuTorch XNNPACK 8da4w + 8-bit embedding | |
| `lfm2_700m_xnnpack_8da4w_e8.pte` (486.5 MB) | |
| - **Source**: LiquidAI/LFM2-700M | |
| - **License**: LFM Open License v1.0 | |
| - **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`) | |
| - **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048, | |
| XNNPACK extended_ops | |
| - **Config**: `llm_params/lfm2_700m_xnnpack_8da4w_e8.yaml` | |
| ## Verification (Mac arm64, 2026-08-21) | |
| `llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per | |
| prompt so no answer is read through the previous one's cache: | |
| | prompt | answer | | |
| |---|---| | |
| | capital of France? | "The capital city of France is Paris." | | |
| | 17 times 4? | "17 times 4 is 68." | | |
| Decode **91.1 tok/s**, from one pass over every model on this shelf with nothing else | |
| running. That matters more than it sounds: the same file measured a quarter of its rate | |
| while an export was running alongside. | |
| Chat template: ChatML, bos 1, eos [7]. | |
| Not measured on a phone. | |
| ## Three things checked before exporting | |
| - **`use_sdpa_with_kv_cache` is on.** Upstream's `qwen3_5` config leaves it off with no | |
| reason given while the equally hybrid `lfm2` config has it on; measured on Qwen3.5-2B in | |
| one run, that is 8.20 tok/s against 16.64. | |
| - **`dim` and `hidden_dim` both divide by the quantizer's group size.** 8da4w only touches a | |
| linear whose in_features divide by it, and skips the rest silently — SmolLM2-135M, which | |
| is 576 wide, came out at 475 MB against fp32's 540 with no warning at all. | |
| - **Every field of the params json is read by the generic path**, via | |
| `convert/check_params_used.py`. SmolLM3 sets `no_rope_layer_interval`, which `ModelArgs` | |
| declares and only the MLX and Qualcomm backends read, and it exports fine and then repeats | |
| a single word forever. | |
| ## Running it | |
| ```bash | |
| python llm_params/gen_static.py \ | |
| --pte lfm2_700m_xnnpack_8da4w_e8.pte \ | |
| --tokenizer tokenizer.json \ | |
| --prompt $'<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n' \ | |
| --eos_ids "[7]" | |
| ``` | |
| The 8-bit embedding needs `from executorch.kernels import quantized` before the program is | |
| loaded, and `portable_lib._load_for_executorch` rather than `executorch.runtime`. Without | |
| that the method will not load at all — `kernel | |
| 'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken | |
| export rather than a runtime missing its kernels. | |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) · | |
| iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples)) | |