README: CoreAIKit's LFM3 support is on the kit's main branch (coreai-kit#83); the Mac bundle's later turns resume from a checkpoint since coreai-models 0.2.9-zoo
132ed7c verified | library_name: coreai | |
| license: other | |
| license_name: lfm1.0 | |
| license_link: https://huggingface.co/mlboydaisuke/LFM3-1.7B-CoreAI/blob/main/LICENSE | |
| base_model: LiquidAI/LFM3-1.7B | |
| tags: | |
| - coreai | |
| - aimodel | |
| - apple-silicon | |
| - on-device | |
| - lfm3 | |
| - hybrid | |
| - reasoning | |
| - tool-calling | |
| pipeline_tag: text-generation | |
| base_model_relation: quantized | |
| # LFM3-1.7B — Apple Core AI (`.aimodel`) | |
| LiquidAI's [LFM3-1.7B](https://huggingface.co/LiquidAI/LFM3-1.7B) converted to Apple's Core AI (iOS 27 / | |
| macOS 27): a 1.7B model that reasons before it answers and calls tools in XML, in two bundles with a 4,096-token | |
| context. The Mac bundle is one int8 `.aimodel` on the `coreai-pipelined` GPU engine that prefills a prompt 64 | |
| tokens at a time; the iPhone 18 Pro bundle is a Neural Engine build with fixed input shapes, compiled in advance for | |
| that phone's chip (`h19p`). | |
| [CoreAIKit](https://github.com/john-rocky/coreai-kit)'s `ChatSession` and Apple's `LanguageModelSession` drove | |
| both in the checks below; CoreAIKit's LFM3 support is on the kit's main branch since `9e9143bc`, not yet in a release | |
| ([Use it](#use-it)). | |
| Source: [`LiquidAI/LFM3-1.7B`](https://huggingface.co/LiquidAI/LFM3-1.7B) at revision | |
| `ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7`. Conversion code, gates and notes: | |
| [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo/tree/main/models/lfm3-1.7b). | |
| ## Bundles | |
| | path | size | for | what is in it | | |
| |---|---:|---|---| | |
| | `gpu-pipelined/lfm3_1_7b_decode_int8hu_block32_sym_pf64/` | 2.72 GB | Mac | `lfm3_1_7b_decode_int8hu_block32_sym_pf64.aimodel`, `metadata.json`, `tokenizer/` (the source's tokenizer files; the chat template is the vendor's with eight rewrites that render the same string under transformers and parse in swift-jinja 2.3.2 and 2.3.6) | | |
| | `ios-h19p/` | 2.16 GB | iPhone 18 Pro | `lfm3_1_7b_static_pal8_tanh_block_c4096.h19p.ane.aimodelc` (compiled for the iPhone 18 Pro's Neural Engine, `h19p`; its `resources.bin` holds 1.48 GB of weights), `metadata.json`, `tokenizer/` (the same four files) | | |
| **Mac.** `int8hu`: the MLP and conv-mixer linears int8 per-block-32, the head untied and int8 per-block-32 | |
| symmetric, the attention projections (q, k, v, o and the gate) and the q / k norms in fp32, everything else fp16. | |
| The `.aimodel` holds two functions over the same weights: `main` takes one token per step and `prefill` takes 64; | |
| the pipelined engine sends the prompt through `prefill` 64 tokens at a time while more than 64 tokens remain, and | |
| the last 1–64 tokens through `main` one at a time. On a Mac, Core AI compiles the `.aimodel` for the GPU the first | |
| time it loads and caches the result. | |
| **iPhone 18 Pro.** The same 24 layers rebuilt the way Apple's iOS Neural Engine path requires (fixed shapes | |
| throughout), with each conv layer's input history kept in its own rows of the KV cache, the sliding window made from | |
| the engine's causal mask, 8-bit palettized weights (k-means, groups of 32; the attention gate fp16) and the tied | |
| head as an int8 table. | |
| It holds contexts of 256 to 4,096 tokens with queries of 8, 16 or 64 tokens, and the compile puts each of its 30 | |
| transformer functions in a Neural Engine region. It is compiled for `h19p` only: no other iPhone has a bundle here. | |
| ## Speed | |
| **Mac** (M4 Max, macOS 27.0), from a 1,024-token prompt: prefill 1,234 tok/s, 0.83 s to the first token, decode | |
| 169.5 tok/s (median of 8 runs, 2026-10-31; the published `.aimodel`, compiled on first load, run by the | |
| coreai-models version CoreAIKit pinned then: the path an app that uses CoreAIKit takes; `llm-benchmark` with 256 generated | |
| tokens, one process per run). Compiled ahead of time for the Mac GPU it is slower: decode 105.5 tok/s and 1.00 s to | |
| the first token in the same window. | |
| In that window, alternated with it on the same 1,024 prompt tokens, MLX (mlx-vlm 0.7.11) decodes LiquidAI's | |
| 8-bit weights at 189.8 tok/s and llama.cpp (b12001) their Q8_0 at 180.2, both faster than this bundle, and their | |
| 4-bit forms faster still (286.9 and 242.1); their bf16 and F16 forms are slower (113.1 and 111.1). All six reach | |
| the first token in 0.23–0.26 s, and every form, this one included, gives transformers bf16's verdicts on the 8 | |
| questions below. What this port adds is a Neural Engine bundle for the iPhone and tool calls through Apple's | |
| `LanguageModelSession`. | |
| **iPhone 18 Pro** (iOS 27.2), a 768-token prompt and 256 generated tokens: prefill 590.6 tok/s, decode 47.17 | |
| tok/s (medians of 6 trials), one turn 6.71 s (768 / prefill + 255 / decode, not timed), peak footprint 2.82 GB, | |
| thermal state nominal at the start and end of every run. In the gate app the first load on the phone took 34.9 s | |
| and later loads 4.4 s or less; a test app built on CoreAIKit (`ChatSession(catalog:)`), with the bundle at its own path, took 32.4 s on its | |
| first load, then 3.7 s and 0.4 s. | |
| ## What was checked | |
| Against transformers 5.20.2 on the same weights. | |
| The Mac bundle, on a Mac (M4 Max, macOS 27.0): | |
| - **Teacher-forced on 17 sequences** (8 questions with thinking off and on, a 1,648-token document) against | |
| transformers fp32, with the `.aimodel` compiled ahead of time for the Mac GPU and the prompts fed the way the | |
| engine feeds them (64-token chunks through `prefill`, the rest one token at a time): top-1 | |
| **2639 / 2682 = 98.40 %** (2,683 positions; the one where transformers' own top-2 gap is under 1e-3 is not | |
| counted); one token at a time throughout: 2637 / 2682. At every chunk boundary the logits agree with the | |
| one-token path to cos ≥ 0.99999. | |
| - **8 questions, thinking off**, greedy: 8 / 8, the same verdicts as transformers bf16. | |
| - **Token gate** (16 greedy tokens vs the fp32 oracle): 16 / 16, also on a 215-token prompt that rides three | |
| chunks; a chat turn stops on `<|im_end|>`. | |
| - **Thinking on**, the same 8 questions, greedy, 3,584-token budget, 3 runs of fresh processes: every run 8 / 8 — | |
| `</think>` closes after 40–237 tokens (median 56) and the answer after it is right (run on the `main`-only | |
| build of the same weights: prompts this short never reach `prefill`, and `main` gives the same values). | |
| - **A tool call**, a tool returning a fresh random value, 3 runs each through CoreAIKit's `KitLanguageModel` and the | |
| zoo's `ZooLanguageModel`: 6 / 6 with the value in the answer verbatim. | |
| - **The bytes published here**: the `.aimodel` is the export re-saved without its source locations (a new | |
| `main.hash`); run again, its teacher-forced values equal the export's at all 2,683 positions, fed in chunks and | |
| one token at a time, and the 8 questions, the token gate and the stop-token check give the same results. | |
| - **The kit's path** (the `.aimodel` specialized on load, as timed above; the teacher-forced pass above was not | |
| repeated on this path): the 8 questions 8 / 8 with the same | |
| replies as the ahead-of-time build, the token gate 16 / 16 on both prompts, and the stop-token turn stops after 6 | |
| tokens (`1+1=2`; transformers fp32 stops after 2, at a step where its own top-2 gap is 0.005). Through | |
| CoreAIKit (`ChatSession` and `KitLanguageModel`), the 8 questions with thinking on (3 launches) and off, a tool | |
| round trip and a 3-turn chat pass. | |
| The iPhone bundle, on an iPhone 18 Pro (iPhone19,2, iOS 27.2), through the fixed-shape (`static-shape`) engine of the | |
| coreai-models version CoreAIKit pinned then: | |
| - **Teacher-forced** over the replies of the same 17 sequences (721 steps, up to 64 a sequence after its prompt) | |
| against transformers fp32: top-1 **692 / 721 = 95.98 %** (bar 93 %; every step's fp32 top-2 gap is at least | |
| 1e-3). The tokens equal those of Apple's own static engine on an iPhone 17 Pro on all 18 prompts, teacher-forced | |
| and generated, and a poisoned fixture fails, as it must. | |
| - **8 questions, thinking off**: 8 / 8, the verdicts of transformers bf16; the stop-token turn stops after 6 tokens. | |
| - **CoreAIKit's catalog path**: `ChatSession(catalog: "lfm3-1.7b")` takes the `ios` variant and loads `ios-h19p/` | |
| (copied into the kit's model store at the path a download would use, so no download ran) on the static-shape | |
| engine. Thinking on, the 8 | |
| questions in 3 fresh launches: 8 / 8 each, `</think>` inside the budget and the reasoning out of the answer. | |
| Thinking off: 8 / 8. A tool round trip through `KitLanguageModel` and `LanguageModelSession`: 3 / 3 with the | |
| value verbatim. With the default configuration, "What is 17 + 25? Answer briefly." gets "17 + 25 = 42.", and a | |
| 3-turn chat recalls its first turn, every turn ending on the stop token. | |
| - With thinking off, 1 of the 8 questions ("How do you say 'thank you' in French?") still opens `<think>` and | |
| reasons for 68 tokens before its answer, which gives "Merci"; transformers bf16 and the Mac bundle answer it | |
| directly. Apple's engine on the iPhone 17 Pro does the same, so this comes from the Neural Engine export itself, | |
| not from CoreAIKit's engine. | |
| The full table, the transcripts and how the fp16 activations set a floor on this model are on the | |
| [zoo card](https://github.com/john-rocky/coreai-model-zoo/tree/main/models/lfm3-1.7b) and in | |
| [the port notes](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/lfm3-1.7b-port.md). | |
| ## Use it | |
| What runs today without CoreAIKit is at the end of this section. CoreAIKit's support for LFM3 — the XML tool | |
| calls, reading a reply that starts inside the thinking block, the thinking switch (`enableThinking`) and the `lfm3-1.7b` catalog | |
| entry — is on the kit's main branch since `9e9143bc` ([coreai-kit#83](https://github.com/john-rocky/coreai-kit/pull/83)), not yet in a CoreAIKit release (0.7.3 does | |
| not support LFM3). The CoreAIKit checks above ran a kit build that has it; with the kit's main branch, on a Mac or an | |
| iPhone 18 Pro: | |
| ```swift | |
| import CoreAIKit | |
| let chat = try await ChatSession(catalog: "lfm3-1.7b") | |
| for try await event in await chat.streamResponse(to: "What is 17 + 25? Answer briefly.") { | |
| switch event { | |
| case .thinking(let delta): print(delta, terminator: "") // the reasoning comes first | |
| case .response(let delta): print(delta, terminator: "") // then the answer | |
| default: break | |
| } | |
| } | |
| ``` | |
| Thinking is on by default (the chat template ends the prompt with `<think>`); `maxResponseTokens` covers the | |
| reasoning and the answer together. `ChatSession.Configuration.enableThinking = false` renders the template's | |
| thinking-off prompt; LiquidAI notes that turning thinking off "can be inconsistent, especially for | |
| reasoning-heavy tasks". | |
| Tool calls come out as XML (`<tool_call>` / `<function=NAME>` / `<parameter=KEY>`). CoreAIKit's `KitLanguageModel` | |
| renders the tools the way the chat template does and parses these calls for Apple's `LanguageModelSession` — the | |
| tool-call checks above. | |
| On an iPhone the kit takes `ios-<arch>/` for the architecture Core AI reports for the phone | |
| (`AIModel.deviceArchitectureName`: `h19p` on the iPhone 18 Pro) and falls back to `ios/`. This repo has | |
| `ios-h19p/` only, so on any other iPhone (the iPhone 17 Pro, which reports `h18p`, included) the kit finds no | |
| bundle and throws `variantNotFound` before it downloads anything. Both apps that ran the iPhone checks carried the | |
| `com.apple.developer.kernel.increased-memory-limit` entitlement (peak footprint 2.82 GB in the gate app, up to 2.94 GB | |
| through `ChatSession`); a run without it was not tried. | |
| Without the kit: the Mac gates ran `llm-runner` built from the zoo's fork of | |
| [apple/coreai-models](https://github.com/apple/coreai-models), whose pipelined engine carries the conv state as an | |
| extra state and finds the `prefill` function by name, with `COREAI_CHUNK_THRESHOLD=1` and | |
| `--inference-engine-variant coreai-pipelined --warmup off`; the iPhone checks ran the `StaticShapeEngine` of that | |
| fork's tag `0.2.8-zoo`, the one CoreAIKit pinned then. The Mac tool-call checks also ran through the zoo's | |
| `ZooLanguageModel` (`swift/Sources/ZooFMProvider` in the zoo; it depends on the coreai-models package, not on | |
| CoreAIKit). | |
| ## Limits | |
| - The engine cannot rewind the Mac bundle's conv state by position. Since coreai-models `0.2.9-zoo` (CoreAIKit's pin | |
| since [coreai-kit#82](https://github.com/john-rocky/coreai-kit/pull/82)), it checkpoints that state at the end of the prompt's last | |
| message, on the 64-token chunk grid, and CoreAIKit's `ChatSession` resumes each later turn from the checkpoint | |
| instead of prefilling the whole history; a history shorter than one chunk is prefilled again (in a three-turn chat | |
| on the kit's main branch, turn 2, after a 32-token first prompt, prefilled all of its 71 tokens, and turn 3 reused 64 | |
| of its 103). The iPhone bundle keeps the conv history in | |
| the KV cache, and in the 3-turn chat its engine reused the earlier turns' prompt tokens (32, then 70). | |
| - On the Mac the engine always runs the last prompt token through the one-token `main` function, so a prompt's | |
| final 1–64 tokens run one at a time at about the decode speed: a 1,024-token prompt runs 15 chunks and then 64 | |
| single steps. | |
| - 4,096-token context in both bundles. | |
| - The iPhone bundle is for the iPhone 18 Pro only; the iPhone 17 Pro has no bundle here. The Mac bundle is not for | |
| iPhones: on an iPhone 18 Pro the `main`-only `.aimodel`, specializing on the phone (device JIT), had not built its | |
| engine 300 s after launch (the cause is not isolated), and on an iPhone 17 Pro the same int8 weights compiled | |
| ahead of time for its GPU decoded at 9.25 tok/s (median of 4 trials at thermal state serious; 10.5 in the first | |
| run, at fair), that compile holding fp16 copies of the int8 linears. | |
| - On a Mac, Core AI named an ahead-of-time compile's | |
| cache entry after its graph and not its weights, so a second compile of the same graph with other weights ran | |
| with the first one's weights (not checked on an iPhone): an app that switches between such compiles should | |
| delete that compile-cache entry first. The Mac bundle is a JIT `.aimodel`, whose cache entry name covers its | |
| weights. | |
| ## License | |
| The weights are under the **LFM Open License v1.0** | |
| ([LICENSE](https://huggingface.co/mlboydaisuke/LFM3-1.7B-CoreAI/blob/main/LICENSE), carried from | |
| [`LiquidAI/LFM3-1.7B`](https://huggingface.co/LiquidAI/LFM3-1.7B)): Apache-2.0-style grants, with §5: | |
| commercial use by a legal entity at or above US$10 M annual revenue is not licensed (the threshold does not | |
| apply to a qualified non-profit's non-commercial or research use). Converted from LiquidAI's weights; not | |
| affiliated with Apple or LiquidAI. | |