Card: measured Performance table (M4 Max + iPhone 17 Pro), verified runnable usage script, accuracy note

README.md

@@ -37,62 +37,136 @@ tags:
 
 # LFM2.5-Encoder-350M — LiteRT
 
-[LiquidAI/LFM2.5-Encoder-350M](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M) converted to **LiteRT** (`.tflite`) for on-device inference. A multilingual (15 languages) bidirectional encoder on the LFM2 hybrid backbone (gated short-convolutions + grouped-query attention) — use it for embeddings, retrieval, classification heads, and masked-token prediction, fully offline on CPU.
+[LiquidAI/LFM2.5-Encoder-350M](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M) converted to **LiteRT** (`.tflite`) for on-device inference. A multilingual (15 languages) bidirectional encoder on the LFM2 hybrid backbone — gated short-convolutions plus grouped-query attention — for embeddings, retrieval, classification heads, and masked-token prediction, fully offline on CPU.
 
-| File | Recipe | Size | |
+## Model description
+
+| File | Recipe | Size | Target |
 |---|---|---|---|
-| `LFM2.5-Encoder-350M_wi8fc.tflite` | int8 dynamic-range (linears + embedding, convs float) | 376 MB | mobile + desktop (iPhone-verified bit-exact) |
+| `LFM2.5-Encoder-350M_wi8fc.tflite` | int8 dynamic-range (linears + embedding, convs float) | 376 MB | mobile + desktop |
 | `LFM2.5-Encoder-350M_fp16.tflite` | fp16 weights, float compute | 713 MB | desktop — XNNPACK's per-signature fp32 unpacking exceeds iPhone memory limits |
 
-## Signatures
-
-All signatures take batch-1, right-padded static shapes: `input_ids` int32 `[1, S]`, `attention_mask` int32 `[1, S]` (1 = real token, 0 = pad).
+All signatures take batch-1, right-padded static shapes: `input_ids` int32 `[1, S]` and `attention_mask` int32 `[1, S]` (1 = real token, 0 = pad).
 
 | Signature | Output |
 |---|---|
 | `encode_64` / `encode_128` / `encode_256` / `encode_512` | `last_hidden_state` float32 `[1, S, 1024]`, zeroed at padded positions |
 | `mlm_128` | masked-LM logits float32 `[1, 128, 65536]` |
 
-Padded positions are fully masked inside the graph (conv path and attention), so the output at valid positions is independent of padding length — `encode_64/128/256` agree bitwise on the same sentence, and match the unpadded PyTorch reference.
+Padded positions are fully masked inside the graph, in both the convolution path and attention, so the output at valid positions does not depend on how much padding follows: `encode_64`, `encode_128` and `encode_256` agree bitwise on the same sentence and match the unpadded PyTorch reference.
 
-## Quality (parity vs PyTorch fp32 reference)
+For a smaller sibling see [LFM2.5-Encoder-230M](https://huggingface.co/litert-community/LFM2.5-Encoder-230M).
 
-16 sentences covering all 15 supported languages; mean-pooled sentence embedding cosine vs the original `Lfm2BidirectionalModel`, and top-5 fill-mask agreement on en/fr/de/ja cloze prompts:
+## How to use
 
-| Variant | Pooled cos (min / mean) | Per-token corr (min) | Fill-mask |
-|---|---|---|---|
-| fp16 | 1.000000 / 1.000000 | 1.000000 | top-5 sets identical (4/4 prompts) |
-| int8 (wi8fc) | 0.996563 / 0.998592 | 0.990448 | top-1 3/4, ≥3/5 top-5 overlap on all |
-
-## Speed (CPU/XNNPACK)
-
-| Variant | Device | encode_128 | encode_512 |
-|---|---|---|---|
-| int8 (wi8fc) | Apple-silicon Mac (all threads) | 46 ms (2765 tok/s) | 123 ms (4180 tok/s) |
-| int8 (wi8fc) | iPhone 17 Pro (6 threads) | 42 ms | 145 ms (~3500 tok/s) |
+**1. Install dependencies**
 
-On the iPhone 17 Pro the int8 model reproduces the Mac outputs **bit-exactly** (cosine 1.000000, max diff 0.0) across all tested languages and signatures; peak footprint ≈1.65 GiB.
+```bash
+pip install ai-edge-litert numpy tokenizers huggingface_hub
+```
 
-## Usage (Python)
+**2. Save the script** below as `embed.py`:
 
 ```python
+#!/usr/bin/env python3
+"""Embed sentences with litert-community/LFM2.5-Encoder-350M and rank them by similarity."""
+import argparse
+
 import numpy as np
 from ai_edge_litert.interpreter import Interpreter
+from huggingface_hub import hf_hub_download
 from tokenizers import Tokenizer
 
-tok = Tokenizer.from_file("tokenizer.json")
-it = Interpreter(model_path="LFM2.5-Encoder-350M_wi8fc.tflite")
-encode = it.get_signature_runner("encode_128")
+REPO = "litert-community/LFM2.5-Encoder-350M"
+MODEL_FILE = "LFM2.5-Encoder-350M_wi8fc.tflite"
+
+
+def embed(runner, tokenizer, text, seq_len):
+    """Mean-pools the encoder states over the real tokens into one vector."""
+    ids = tokenizer.encode(text).ids
+    if len(ids) > seq_len:
+        raise SystemExit(f"{len(ids)} tokens exceed --seq-len {seq_len}")
+    input_ids = np.zeros((1, seq_len), np.int32)
+    attention_mask = np.zeros((1, seq_len), np.int32)
+    input_ids[0, : len(ids)] = ids
+    attention_mask[0, : len(ids)] = 1
+    states = list(runner(input_ids=input_ids, attention_mask=attention_mask).values())[0]
+    vector = states[0, : len(ids)].mean(axis=0)
+    return vector / np.linalg.norm(vector)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--query", required=True, help="The sentence to match.")
+    parser.add_argument("--candidate", action="append", required=True,
+                        help="A candidate sentence, repeatable.")
+    parser.add_argument("--seq-len", type=int, default=128,
+                        choices=[64, 128, 256, 512])
+    parser.add_argument("--threads", type=int, default=8)
+    args = parser.parse_args()
+
+    model_path = hf_hub_download(REPO, MODEL_FILE)
+    tokenizer = Tokenizer.from_file(hf_hub_download(REPO, "tokenizer.json"))
+    interpreter = Interpreter(model_path=model_path, num_threads=args.threads)
+    runner = interpreter.get_signature_runner(f"encode_{args.seq_len}")
+
+    query = embed(runner, tokenizer, args.query, args.seq_len)
+    scored = [(float(query @ embed(runner, tokenizer, c, args.seq_len)), c)
+              for c in args.candidate]
+    for score, text in sorted(scored, reverse=True):
+        print(f"{score:6.3f}  {text}")
+
+
+if __name__ == "__main__":
+    main()
+```
 
-ids = tok.encode("On-device embeddings, private and fast.").ids
-x = np.zeros((1, 128), np.int32); m = np.zeros((1, 128), np.int32)
-x[0, :len(ids)] = ids; m[0, :len(ids)] = 1
+**3. Run it**
 
-h = list(encode(input_ids=x, attention_mask=m).values())[0]  # [1, 128, 1024]
-emb = h[0, :len(ids)].mean(axis=0)                           # sentence embedding
+```bash
+python embed.py --query "What is your refund policy?" \
+  --candidate "Our refund policy allows 30 days" \
+  --candidate "Steps to recover a forgotten login" \
+  --candidate "The weather in Osaka is mild in spring"
 ```
 
-For masked-token prediction use the `mlm_128` signature and read the logits at the `[MASK]` position. On Android/iOS use the LiteRT runtime's SignatureRunner APIs with the same signature names; the tokenizer is the standard Hugging Face `tokenizer.json` (works with the `tokenizers` libraries for Rust/Swift/Kotlin).
+```
+ 0.604  Our refund policy allows 30 days
+ 0.579  The weather in Osaka is mild in spring
+ 0.570  Steps to recover a forgotten login
+```
+
+Mean-pooling the raw encoder states is the simplest sentence representation and is what the numbers above use; for retrieval at quality you would normally train a pooling head or fine-tune on your own pairs. For masked-token prediction use the `mlm_128` signature and read the logits at the `[MASK]` position.
+
+On Android/iOS use the LiteRT runtime's SignatureRunner APIs with the same signature names; the tokenizer is the standard Hugging Face `tokenizer.json`, which the Rust/Swift/Kotlin `tokenizers` bindings all read.
+
+## Performance
+
+int8 (`wi8fc`) file, CPU only.
+
+| Device | Threads | `encode_128` | `encode_512` | `mlm_128` |
+|---|---|---|---|---|
+| Apple M4 Max (macOS) | 8 | 36.5 ms | 111.2 ms | 44.7 ms |
+| iPhone 17 Pro | 6 | 42 ms | 145 ms | 73 ms |
+
+Mac figures are the median of 20 warm runs (`ai-edge-litert` 2.1.6, XNNPACK, otherwise idle machine). iPhone figures come from the on-device gate (TFLite C API + SignatureRunner + XNNPACK); each is the last of three consecutive measurements, not a median.
+
+**Budget for one slow first call.** The first inference after loading pays a one-time graph preparation. On the Mac that first call took 908 ms against a 36.5 ms steady state; on the iPhone the three consecutive `encode_128` measurements were 102, 51 and 42 ms, and the three `encode_512` measurements were 229, 152 and 145 ms. Model load was 1.48 s on the iPhone, with a peak footprint of about 1.65 GiB.
+
+Those three consecutive values are **not** a language effect — the signatures are fixed-shape, so every input of a given signature costs the same. Measured warm on the Mac with the run order reversed, one signature takes 36.4 / 37.3 / 36.6 ms on English, Japanese and Arabic sentences of 17, 21 and 27 tokens.
+
+Thread count matters more than anything else here: at the interpreter default this model measures 52.4 ms and 227.1 ms for `encode_128` and `encode_512`, against 36.5 ms and 111.2 ms at 8 threads.
+
+### Accuracy note
+
+Parity against the PyTorch fp32 reference over 16 sentences covering all 15 supported languages — mean-pooled sentence-embedding cosine against the original `Lfm2BidirectionalModel`, plus top-5 fill-mask agreement on English, French, German and Japanese cloze prompts:
+
+| Variant | Pooled cosine (min / mean) | Per-token correlation (min) | Fill-mask |
+|---|---|---|---|
+| fp16 | 1.000000 / 1.000000 | 1.000000 | top-5 sets identical (4/4 prompts) |
+| int8 (`wi8fc`) | 0.996563 / 0.998592 | 0.990448 | top-1 on 3/4, at least 3/5 top-5 overlap on all |
+
+On the iPhone 17 Pro the int8 file reproduces the Mac outputs **bit-exactly** — cosine 1.000000, max absolute difference 0.0 — across every tested language and signature.
 
 ## License