| --- |
| license: apache-2.0 |
| base_model: google/gemma-4-E2B-it |
| tags: |
| - coreml |
| - apple-neural-engine |
| - decision-model |
| - d1a |
| - 4-bit |
| - experimental |
| library_name: coreml |
| pipeline_tag: text-classification |
| --- |
| |
| <p align="center"><img src="https://huggingface.co/JohnP1/d1a-e2b-it-coreml-4bit/resolve/main/d1a-avatar.png" width="200" alt="D1A avatar: an anime girl with silver-white and dark purple hair, round glasses and headphones on a round #D1A badge"></p> |
|
|
| # D1A-E2B-it v0.6 · Core ML int4 (text, ≤ 512 tokens, experimental) |
|
|
| **A small decision model for the Apple Neural Engine: give it a document and a few questions, and it picks the answers.** Each question |
| is yes/no, multiple choice or a score; D1A returns a calibrated probability for every option in one forward pass, with no |
| generated text, so every answer is a valid option. |
|
|
| **This is an experimental build for on-device apps that already load Gemma 4 E2B-it**: the Core ML conversion of |
| [JohnP1/d1a-e2b-it](https://huggingface.co/JohnP1/d1a-e2b-it) v0.6, D1A-E2B v0.6's recipe on the |
| [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) trunk. Where it has been measured (on the MLX 8-bit build) |
| it is below D1A-E2B v0.6: PR severity on 487 newer PRs 72.9% against 77.2% (−4.3 [−7.8, −1.0]), change type 87.3% against |
| 87.5%, and media 21/28 against 25/28 agreement with E4B v0.6. E4B v0.6 remains the PR labeler. For anything but an app that |
| already ships Gemma 4 E2B-it, use [JohnP1/d1a-e2b-coreml](https://huggingface.co/JohnP1/d1a-e2b-coreml) v0.6. |
|
|
| This is the 4-bit Core ML build: **text only, documents up to 512 tokens**, int4 weights (per block of 32), 2.6 GB (1.0 GB chunks + 1.6 GB embeddings). |
| The int8 build is [JohnP1/d1a-e2b-it-coreml](https://huggingface.co/JohnP1/d1a-e2b-it-coreml). The conversion is the one |
| described in jonpol01/d1a#224 for E2B v0.6, with block-wise 4-bit weights (blocks of 32) in place of int8 (blocks of 64); |
| the embedding tables on the host are the same 8-bit and 4-bit MLX tables as the int8 build. Block-wise int4 needs |
| **iOS 18 / macOS 15**. |
|
|
| ## Status |
|
|
| - **Verified on an iPhone 17 Pro (iOS 27)** in an iOS app on 2026-10-13, with the same 28 requests: 0 of 42 answers differ |
| from the Mac Core ML run (largest probability change 0.016), with all five chunks in one process. |
| - **GPU** (`.cpuAndGPU`): 0.66 s per request (p50), 1.2–1.5 GB, 20 s to load (warm). |
| - **Neural Engine** (`.cpuAndNeuralEngine`): 1.19 s, 3.85 GB (the Neural Engine's compiled programs count toward the |
| app's memory), 134 s to load (warm), about 5.5 min the first time (on-device compile). **On this phone, use the GPU.** |
| - **Photos:** Gemma 4 E2B-it's own vision encoder, converted to Core ML separately (fixed 672×912 → 266 image tokens; not |
| in this repository yet), ran with these chunks on the iPhone's GPU. |
| - 6 demo photos: 0.52 s per request including the encoder, 0 of 12 answers differ from the Mac. |
| - With the encoder on the Neural Engine as well (six Neural Engine programs in one process), the app hung. |
| - **Mac check** on an Apple M1 Max, 28 real requests of up to 512 tokens (42 questions), against the fp32 PyTorch rebuild of |
| the 8-bit export it was converted from: p50 16.0 s, p95 55.6 s per request (28 requests), 338 s to load (first, compiled); 4 of 42 answers change (2 of 40 clear-cut ones, where the reference's top answer leads by more than 0.05); largest probability change 0.326. |
| - On M1-class Macs, E2B v0.6's int8 chunks ran on the CPU (the M1's Neural Engine did not take them). Whether these int4 |
| chunks do the same is not checked, so the M1 Max timing above may not be a Neural Engine timing. E2B v0.6's int8 build |
| took 0.96 s per request on an M4's Neural Engine; this build has not been timed on an M4. |
| - **The model behind it is experimental**: its pre-registered bar (jonpol01/d1a#227, Bar B) failed on severity and media, and |
| most suites (transfer-v4, hard, devtools, documents, decision-v7, JGLUE) are not measured yet. It is published by the |
| owner's decision of 2026-10-12 for the iOS proof of concept. |
| - **Not included:** photo, voice and video (their encoders are separate models, not converted), and inputs longer than 512 |
| tokens. |
| |
| ## `<bos>`: encode with it |
|
|
| The Gemma 4 E2B-it tokenizer adds no `<bos>`, but this model was trained with one at the start of every record. d1a from |
| jonpol01/d1a#261 on (merged as `b4b0f989`) puts it back (bos_token_id 2; `d1a_config.json` records `leading_ids: [2]`). An app that encodes requests |
| itself must prepend id 2 before right-padding to 512. |
|
|
| ## Run it |
|
|
| Needs d1a with jonpol01/d1a#261: |
|
|
| ```bash |
| pip install coremltools numpy torch transformers safetensors "d1a @ git+https://github.com/jonpol01/d1a" |
| hf download JohnP1/d1a-e2b-it-coreml-4bit --revision v0.6 --local-dir d1a-e2b-it-coreml-4bit |
| python d1a-e2b-it-coreml-4bit/reference/run.py d1a-e2b-it-coreml-4bit request.json |
| ``` |
|
|
| `request.json` is a System One request, for example: |
|
|
| ```json |
| {"state": "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.", |
| "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"returns": null, "shipping": null, "billing": null}}, |
| "urgent": {"type": "noul", "instructions": "Does this need a reply today?"}}} |
| ``` |
|
|
| The first run compiles the chunks into `coreml/.compiled/` (a few minutes); later runs load them. |
|
|
| ## Files |
|
|
| | Path | What | |
| |---|---| |
| | `coreml/L512_<a>-<b>.mlpackage` | 5 chunks of the 35-layer text model, int4 weights (per block of 32). Chunk 7-15 outputs the shared keys and values that chunks 15-35 read. | |
| | `embeddings.safetensors` | the token and per-layer embedding tables (MLX affine-quantized, from JohnP1/d1a-e2b-it-mlx-q8@v0.6), read row by row on the host | |
| | `head.safetensors`, `d1a_config.json` | the pointer head, the temperature (1.41), the `<bos>`, pad and delimiter ids | |
| | `config.json`, `tokenizer.json`, `tokenizer_config.json` | the Gemma 4 E2B-it model config and tokenizer | |
| | `io.json` | each chunk's input names | |
| | `reference/` | Python reference: `run.py` (one request), `workers.py` (one process per chunk), `d1a_coreml.py` (host-side embeddings and masks), `pipeline.py` (all chunks in one process) | |
|
|
| ## How a request runs |
|
|
| 1. Encode it with D1A's layout (`d1a.core.encoding.encode`, d1a with #261, so it starts with `<bos>`, id 2) and right-pad it |
| to 512 tokens with the pad token. |
| 2. Look up the token embeddings (scaled by √hidden) and the per-layer embeddings (scaled by √256) on the host. |
| 3. Build the block-causal masks (full and sliding), clamped to −1e4. |
| 4. Run the chunks in order, each in its own process. Several large Neural Engine programs in one process can crash the |
| runtime. |
| 5. Read the hidden states at each question's `<decide>` and `</opt>` positions, score them with the head, divide by the |
| temperature, and take the softmax. |
|
|
| macOS 15 / iOS 18 or later (block-wise int4 weights). |
|
|
| ## D1A-E2B-it builds |
|
|
| The same model in four builds. Mac checks on an Apple M1 Max, the same 28 requests (42 questions, all ≤ 512 tokens), each |
| against the fp32 PyTorch rebuild of the 8-bit export: |
|
|
| | Build | Repository | Runs on | Size | Mac check | |
| |---|---|---|---|---| |
| | MLX 8-bit | JohnP1/d1a-e2b-it-mlx-q8 | Mac GPU | 3.5 GB + 0.9 GB media | on its card | |
| | MLX 4-bit | JohnP1/d1a-e2b-it-mlx-q4 | Mac GPU | 2.5 GB + 0.9 GB media | on its card | |
| | Core ML int8 | JohnP1/d1a-e2b-it-coreml | Neural Engine (CPU on M1) | 3.5 GB (1.8 GB chunks + 1.6 GB embeddings) | on its card | |
| | **Core ML int4** | **JohnP1/d1a-e2b-it-coreml-4bit** | Neural Engine (M1: not checked) | 2.6 GB (1.0 GB chunks + 1.6 GB embeddings) | p50 16.0 s, p95 55.6 s per request (28 requests), 338 s to load (first, compiled); 4 of 42 answers change (2 of 40 clear-cut ones, where the reference's top answer leads by more than 0.05); largest probability change 0.326. | |
|
|
| ## License |
|
|
| Apache-2.0. Base model: Gemma 4 by Google (Apache-2.0). D1A is built on Kev by Jared Palmer (Apache-2.0). |
|
|
