iPhone peak memory and prefill at 128K and 256K context
Congratulations on the launch. The card gives a 256K context window and about 1 GB for 4-bit weights, while the iPhone 17 Pro edge chart uses 32,768 tokens, batch size 1, and 100 output tokens, and reports 1.90 GB of peak process memory.
Could you share peak process memory, including KV/cache and runtime, plus time to first token for 128K and 256K prompts on iPhone 17 Pro with the same GGUF setup? If the full context does not fit on a phone, the largest usable context before memory pressure would be helpful too.
The pinned config has six full-attention layers, five 256-token sliding-attention layers, four KV heads and 128 dimensions per head. Assuming f16 K and V with no cache compression, that implies roughly 0.38 GiB of attention cache at 32K, 1.50 GiB at 128K and 3.00 GiB at 256K, before weights and other buffers. The actual llama.cpp allocation could differ, especially with quantized KV.
These measured figures would help builders plan local-agent workloads around the advertised context window.
Thanks for the question, and for the careful cache math. Your figures are correct. The six full-attention layers use 12 KiB per token for f16 K and V. That gives 0.375 GiB at 32K, 1.5 GiB at 128K and 3.0 GiB at 256K. The five sliding-window layers keep only 256 tokens each, so they add about 2.5 MiB. The conv layers add almost nothing.
We measured long-context performance on an iPhone 17 Pro (A19 Pro) with the same setup as the model card:
- Model: LFM3-1.7B Q4_0 GGUF
- Batch size 1 and 100 output tokens
- All layers offloaded to Metal, with an f16 KV cache
| Prompt length | Prefill | Time to first token | Peak process memory | Decode |
|---|---|---|---|---|
| 32K | ~1,050 tok/s | ~30 s | ~1.9 GB | ~44 tok/s |
| 128K | ~480 tok/s | ~4.5 min | ~3.3 GB | ~22 tok/s |
| 256K | ~270 tok/s | ~16 min | ~5.1 GB | — |
The full 256K context fits on the phone. Peak memory stays near 5 GB, and memory grows linearly at about 14 KB per token. That rate is close to your KV estimate plus compute buffers. At 256K, memory is not the limit. Prefill time is the limit: a cold 256K prompt takes about 16 minutes to process.
For local agents, we recommend that you keep the prompt prefix stable and reuse the KV cache across turns. Do not process the full context again on each turn. With cache reuse, each turn processes only the new tokens.