README.md

---
library_name: coreai
license: apache-2.0
base_model: ibm-granite/granite-embedding-97m-multilingual-r2
tags:
  - coreai
  - sentence-similarity
  - feature-extraction
  - apple-silicon
  - on-device
  - modernbert
language:
  - multilingual
  - ja
  - en
pipeline_tag: sentence-similarity
---

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's `coreai-torch` (LLMs: `coreai.llm.export`) into `.aimodel` bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol ([apple-silicon-llm-bench](https://github.com/john-rocky/apple-silicon-llm-bench), macOS 27 beta, 2026-06).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->
This model has no row on [DeviceMark](https://devicemark.github.io/), the on-device LLM leaderboard.
<!-- gen-cards:devicemark end -->

# Granite-Embedding-97M-Multilingual-R2 — Core AI export

Zoo card, recipe and gate transcript: [coreai-model-zoo/models/granite-embedding-97m](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/README.md).

IBM's 97M-parameter **multilingual text embedder** — a ModernBERT encoder, 384-d CLS-pooled
unit vectors, Japanese and English among its languages — as a static `.aimodel` for macOS 27
and iOS 27, with bundles compiled ahead of time for the iPhone 17 Pro beside it.
[`ibm-granite/granite-embedding-97m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2)
(Apache-2.0, revision `835ad1408…`) is the **smallest embedder in this catalog** (390 MB fp32,
against 1.2 GB for EmbeddingGemma-300m and 1.1 GB for Qwen3-Embedding-0.6B) and its **first
encoder-architecture one** — every other embedder here is a causal decoder run as an encoder.
Its retrieval quality relative to those three was **not** measured here: the fixture set below
is a parity instrument (35 texts, 4 queries, 12 documents), not a benchmark.

**This is an encoder, not a generator** — one forward over the right-padded grid returns one
unit vector. No autoregressive loop, no KV cache, no LM head. It runs as a plain `.aimodel`
through raw `AIModel.run` (like the vision encoders), not the pipelined generate engine.

Architecture (`model_type: modernbert`): 12 layers, hidden 384, 12 heads × 32, GLU MLP 1536
(SiLU), vocabulary 180,000, biasless everything (attention, MLP, LayerNorm ε 1e-5). Global
attention at layers **0, 3, 6, 9** (RoPE θ 150,000); the other eight are **local**, a sliding
window of inclusive radius 64 (129 keys per interior query, RoPE θ 160,000). Layer 0 has no
attention pre-norm (the embedding LayerNorm serves). Pooling is CLS → L2 normalize, both in the
graph.

## Graph contract

```
input  "input_ids"       [1, S]    int32   right-padded to the grid S with 179935
input  "attention_mask"  [1, S]    int32   1 over real tokens, 0 over padding
output "embedding"       [1, 384]  fp32    CLS-pooled, L2-normalized
S = 128 or 512 (export-time choice); batch = 1
```

**Host recipe** — the tokenizer is the whole contract, and the stock one is not enough:
- **No prefix, no stripping, no normalization.** Query and document prompts are both empty in
  the checkpoint. Raw whitespace is kept: sentence-transformers strips text before tokenizing,
  the upstream README's `AutoTokenizer` path does not, and the two disagree on `"  東京駅から…\n"`.
  The reference is the raw path.
- Tokenize with the pinned `tokenizer.json`: regex `Split(Isolated)` → `ByteLevel` (no prefix
  space) → byte BPE with **`ignore_merges = true`** (a whole pre-token that is in the vocabulary
  wins; ` ક` is token 2999, not three). A BPE that ignores the flag tokenizes differently.
- Truncate the **body to S−2**, then wrap: `[CLS 179934] body… [SEP 179938]`, right-pad with
  **PAD 179935** and mask 0. Truncating after adding the specials loses SEP; padding with 0 is a
  different token. Both are silent.
- Similarity = dot product (unit vectors). Dimension truncation is not a property of this model.

[`conversion/granite_embedding/_granite_tokenizer.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/granite_embedding/_granite_tokenizer.py) is that recipe with no HF import, and
`host/GraniteTokenizer.swift` in this repo the same recipe in Foundation-only Swift; the gate
holds both to `AutoTokenizer` exactly (ids and masks) over **681 texts × 2 grids = 1,362 cases**
including every added token in five boundary contexts, and proves four mutations are caught
(pad 0 / lose SEP / strip / `ignore_merges=false`).

## Measured

**iPhone 17 Pro** (iPhone18,1), iOS 27.0 build **24A437**, the compiled `h18p` bundles loaded by
the native `AIModel` loader, GPU-preferred (MPSGraph/Metal plan). Every row: 35 HF texts, the
gate below, 105 warm samples, thermal state fair before and after, caches retained (so "first"
is process-first, not cache-cold). Peak footprint is the whole app process, tokenizer and file
hashing included. Measured 2026-09-19.

| Variant | S | Gate | Min cosine vs HF | Max \|err\| | Load | First after load | **Warm median** | Peak footprint |
|---|---:|---|---:|---:|---:|---:|---:|---:|
| fp32 | 128 | 35/35 | 0.999999999999407 | 1.97e-7 | 81 ms | 23.1 ms | **5.54 ms** | 640 MB |
| fp32 | 512 | 35/35 | 0.999999999999486 | 2.38e-7 | 586 ms | 39.9 ms | **20.99 ms** | 640 MB |
| w8 / fp32 table | 128 | 35/35 | 0.999410 | 5.67e-3 | 61 ms | 25.3 ms | 6.64 ms | 555 MB |
| w8 / fp32 table | 512 | 35/35 | 0.999410 | 5.67e-3 | 447 ms | 138.7 ms | 23.07 ms | 553 MB |

Each row matched 4/4 retrieval top-1s with 0 clear-pair flips and 0 repeat drift.

**Mac** (M4 Max, Mac16,9), macOS 27.0 build 26A428, the JIT `.aimodel`, GPU-preferred, fp32.
The driver refused to run while any foreign accelerator job was present; 105 warm samples.

| S | Gate | Min cosine vs HF | Max \|err\| | Load | First after load | **Warm median** |
|---:|---|---:|---:|---:|---:|---:|
| 128 | 35/35 | 0.99999999999967 | 2.98e-7 | 481 ms | 642 ms | **4.14 ms** |
| 512 | 35/35 | 0.99999999999887 | 2.98e-7 | 470 ms | 264 ms | **4.89 ms** |

The Mac h16c AOT twin also passed 70/70 (same numerics), but its timings were taken with another
lane's GPU evaluation running and are not reported. The w8 variant on Mac is gated on **CPU
only** (min cosine 0.999410, max |err| 5.67e-3, ranking exact); Mac GPU for w8 was not run.

**iPhone 18 Pro** (iPhone19,2, h19p), iOS 27.0 build **24A437**, the JIT `.aimodel` bundles in `ios/`
(the same files as `macos/`), loaded 2026-09-26 by the zoo's DecideGate app in its load-only mode,
GPU-preferred, without the increased-memory entitlement. Each first load was the first after a fresh
install of the app. The call is one run on all-zero inputs. Each first load wrote a specialization of
about the bundle's size into the app container, and the load after a relaunch reused it. One measurement
per graph
([knowledge/jit-distribution.md](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/jit-distribution.md)).

| JIT bundle in `ios/` | MB | first load | first call | load after relaunch |
|---|---:|---:|---:|---:|
| `fp32-s128/granite97m_fp32_s128_bound.aimodel` | 390 | 1.01 s | 619 ms | 0.26 s |
| `fp32-s512/granite97m_fp32_s512_bound.aimodel` | 390 | 0.40 s | 88 ms | 0.26 s |
| `w8-fp32table-s128/granite97m_w8_fp32table_s128.aimodel` | 305 | 0.63 s | 132 ms | 0.21 s |
| `w8-fp32table-s512/granite97m_w8_fp32table_s512.aimodel` | 306 | 0.33 s | 93 ms | 0.21 s |

The iPhone gate above (the iPhone 17 Pro rows) ran the `h18p` export, now in `ios-h18p/`. The JIT IR in
`ios/` was only loaded and called once on the iPhone 18 Pro. That call returned a 384-value float32
embedding with no non-finite values; its parity with the reference was not re-measured.

The fixed grid computes every position, so pick the smallest grid that covers the text: S=128
for queries and short notes, S=512 for passages. **fp32 is the default.** w8 is a storage
option only — 22% smaller, not faster here — because the 180,000×384 fp32 vocabulary table is
276 MB of the bundle and palettization touches the 48 linear weights alone.

## Numerics gate

One gate at every stage, the oracle being official HF eager CPU fp32 (transformers 4.57.6):
per text cosine ≥ 0.999, max element error ≤ 0.02, L2-norm error ≤ 0.002; per query exact
top-1 over the 12 documents, retrieval-score error ≤ 0.01, and no inversion of any document
pair the oracle separates by ≥ 0.001; repeat drift ≤ 1e-6. A wrong-pairing control (every vector
matched to the wrong text) must FAIL.

- **Authoring** (`gate_granite_authoring.py`): the re-authored graph against every one of the 13
  saved hidden states, max |err| ≤ **1e-4** at fp32, both grids. Five mutations must trip it:
  all-global, all-local, ignore-padding and mean-pooling fail the embedding gate; a local radius of
  **63 instead of 64** passes the embedding gate (cos 0.99995) and fails only the layer gate —
  which is why the layer gate exists. Whole-model **fp16 fails** this layer gate on both grids.
- **Export**: the torch-exported, decomposed graph is gated before conversion, on both grids.
- **Runtime**: Mac CPU and GPU (JIT), Mac h16c AOT, iPhone h18p AOT — the tables above; iPhone 18 Pro
  JIT: load and one zero-input call only.
- **w8**: the same gate at prepared, finalized and decomposed stages, 48 `lut_to_dense` ops
  counted, palettes hashed; the iOS w8 export reuses the Mac palettes byte for byte.

[`models/granite-embedding-97m/gate-granite-embedding-97m.json`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/gate-granite-embedding-97m.json) in the zoo is the transcript: the eight runtime rows
(min cosine, max error, retrieval, timings, device/OS build), the tokenizer gate and the
authoring gate, each with the sha256 of the full record it summarizes.

## ⬇️ Bundle

This repo — one folder per variant, each self-contained: the bundle, `tokenizer/`, `reference.json` (the
35 HF fixtures with ids, masks and embeddings — the parity test) and `provenance/` (export
manifest with per-file sha256, the runtime gate record). `coreai-kit.json` at the root maps
platform → folder.

| Folder | Platform | Format | Bundle | Bytes |
|---|---|---|---|---:|
| `macos/fp32-s512/` **(default)** | macOS 27 | JIT `.aimodel` | `granite97m_fp32_s512_bound.aimodel` | 390,431,506 |
| `macos/fp32-s128/` | macOS 27 | JIT `.aimodel` | `granite97m_fp32_s128_bound.aimodel` | 389,989,146 |
| `ios/fp32-s512/` **(default)** | iOS 27 | JIT `.aimodel` | `granite97m_fp32_s512_bound.aimodel` | 390,431,506 |
| `ios/fp32-s128/` | iOS 27 | JIT `.aimodel` | `granite97m_fp32_s128_bound.aimodel` | 389,989,146 |
| `macos/w8-fp32table-s512/` | macOS 27 (CPU-gated) | JIT `.aimodel` | `granite97m_w8_fp32table_s512.aimodel` | 305,569,358 |
| `macos/w8-fp32table-s128/` | macOS 27 (CPU-gated) | JIT `.aimodel` | `granite97m_w8_fp32table_s128.aimodel` | 305,126,985 |
| `ios/w8-fp32table-s512/` | iOS 27 | JIT `.aimodel` | `granite97m_w8_fp32table_s512.aimodel` | 305,569,358 |
| `ios/w8-fp32table-s128/` | iOS 27 | JIT `.aimodel` | `granite97m_w8_fp32table_s128.aimodel` | 305,126,985 |
| `ios-h18p/fp32-s512/` | iOS 27, **h18p only** | AOT `.aimodelc` | `granite97m_fp32_s512_bound.h18p.aimodelc` | 390,308,788 |
| `ios-h18p/fp32-s128/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_fp32_s128_bound.h18p.aimodelc` | 390,081,410 |
| `ios-h18p/w8-fp32table-s512/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_w8_fp32table_s512_r02.h18p.aimodelc` | 305,479,184 |
| `ios-h18p/w8-fp32table-s128/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_w8_fp32table_s128_r02.h18p.aimodelc` | 305,251,774 |

`ios/` holds the same JIT bundles as `macos/`; every iPhone generation specializes them on its first
load. The `ios-h18p/` bundles moved there from `ios/` in revision `a27dc73e` (2026-09-26). They are
compiled for one device architecture (`h18p`, the iPhone 17 Pro) with
`xcrun coreai-build compile --platform iOS --min-deployment-version 27.0 --preferred-compute gpu
--architecture h18p` (coreai-build 3600.83.1), from a separate iOS export whose IR is reproducible
from the recipe but not shipped. The iPhone 18 Pro refuses an h18p bundle with
`incompatibleCompiledAssetArchitecture`. **Never load an `ios-h18p/` bundle on a Mac.**

Convert yourself: [`conversion/granite_embedding/`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/granite_embedding/README.md)
— five staged scripts; [`recipe.toml`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/recipe.toml) names the commands.

## CoreAIKit (Swift)

**Not enrolled** in the kit catalog. The kit's `TextEmbedder` pads with 0, truncates after adding
the special tokens (losing SEP), applies its own BPE without `ignore_merges`, discovers a single
`.aimodel`, and has no grid / architecture selection — every one of those is wrong for this
model. Running it today means: the Swift tokenizer from this repo's `host/` folder, a fixed
grid, `AIModel` on the platform's folder. Enrolling it needs a `textEmbedding` driver that takes
the pad id, a SEP-preserving truncation, a per-platform variant path and an AOT-aware loader —
tracked as maintainer work, not a blocker on the bundle.

## The port in one lesson: gate the layers, not just the vector

ModernBERT's alternating local/global attention is the whole risk. The config says
`local_attention: 128`; the executed window is inclusive `|i − j| ≤ 64` — 129 keys — and a
window of 63 reproduces the final embedding to cos 0.99995 while every hidden state past layer 1
is wrong. Only a per-layer oracle catches it. Three more things the raw checkpoint settles that
the modeling file hides: layer 0 has no attention norm (adding one loads a missing weight),
the two RoPE thetas are per-layer-kind, and the CLS/L2 head needs an explicit `clamp_min`
epsilon because the converter's `F.normalize` decomposition drops it.

## License and limits

Apache-2.0 at the pinned upstream revision; this repo carries IBM's unmodified card as
`UPSTREAM_README.md` and a `LICENSE-NOTE.md` listing the changes (static graph, in-graph
pooling, optional w8 palettes, h18p compile). Not tested: phones other than the iPhone 17 Pro (the
h18p gate) and the iPhone 18 Pro (JIT load and one call), other OS builds, the JIT bundles' embedding
parity on an iPhone, the Mac GPU with w8, the Neural Engine, dynamic or batched shapes, S > 512, languages beyond the JA/EN
fixtures, retrieval quality on a benchmark, sustained thermals, true cache-cold load.