JohnP1/d1a-e2b-it-coreml-4bit

D1A-E2B-it v0.6 · Core ML int4 (text, ≤ 512 tokens, experimental)

A small decision model for the Apple Neural Engine: give it a document and a few questions, and it picks the answers. Each question is yes/no, multiple choice or a score; D1A returns a calibrated probability for every option in one forward pass, with no generated text, so every answer is a valid option.

This is an experimental build for on-device apps that already load Gemma 4 E2B-it: the Core ML conversion of JohnP1/d1a-e2b-it v0.6, D1A-E2B v0.6's recipe on the google/gemma-4-E2B-it trunk. Where it has been measured (on the MLX 8-bit build) it is below D1A-E2B v0.6: PR severity on 487 newer PRs 72.9% against 77.2% (−4.3 [−7.8, −1.0]), change type 87.3% against 87.5%, and media 21/28 against 25/28 agreement with E4B v0.6. E4B v0.6 remains the PR labeler. For anything but an app that already ships Gemma 4 E2B-it, use JohnP1/d1a-e2b-coreml v0.6.

This is the 4-bit Core ML build: text only, documents up to 512 tokens, int4 weights (per block of 32), 2.6 GB (1.0 GB chunks + 1.6 GB embeddings). The int8 build is JohnP1/d1a-e2b-it-coreml. The conversion is the one described in jonpol01/d1a#224 for E2B v0.6, with block-wise 4-bit weights (blocks of 32) in place of int8 (blocks of 64); the embedding tables on the host are the same 8-bit and 4-bit MLX tables as the int8 build. Block-wise int4 needs iOS 18 / macOS 15.

Status

• Verified on an iPhone 17 Pro (iOS 27) in the Mieru app on 2026-10-13, with the same 28 requests: 0 of 42 answers differ from the Mac Core ML run (largest probability change 0.016), with all five chunks in one process.
  • GPU (.cpuAndGPU): 0.66 s per request (p50), 1.2–1.5 GB, 20 s to load (warm).
  • Neural Engine (.cpuAndNeuralEngine): 1.19 s, 3.85 GB (the Neural Engine's compiled programs count toward the app's memory), 134 s to load (warm), about 5.5 min the first time (on-device compile). On this phone, use the GPU.
• Photos: Gemma 4 E2B-it's own vision encoder, converted to Core ML separately (fixed 672×912 → 266 image tokens; not in this repository yet), ran with these chunks on the iPhone's GPU.
  • 6 demo photos: 0.52 s per request including the encoder, 0 of 12 answers differ from the Mac.
  • With the encoder on the Neural Engine as well (six Neural Engine programs in one process), the app hung.
• Mac check on an Apple M1 Max, 28 real requests of up to 512 tokens (42 questions), against the fp32 PyTorch rebuild of the 8-bit export it was converted from: p50 16.0 s, p95 55.6 s per request (28 requests), 338 s to load (first, compiled); 4 of 42 answers change (2 of 40 clear-cut ones, where the reference's top answer leads by more than 0.05); largest probability change 0.326.
• On M1-class Macs, E2B v0.6's int8 chunks ran on the CPU (the M1's Neural Engine did not take them). Whether these int4 chunks do the same is not checked, so the M1 Max timing above may not be a Neural Engine timing. E2B v0.6's int8 build took 0.96 s per request on an M4's Neural Engine; this build has not been timed on an M4.
• The model behind it is experimental: its pre-registered bar (jonpol01/d1a#227, Bar B) failed on severity and media, and most suites (transfer-v4, hard, devtools, documents, decision-v7, JGLUE) are not measured yet. It is published by the owner's decision of 2026-10-12 for the iOS proof of concept.
• Not included: photo, voice and video (their encoders are separate models, not converted), and inputs longer than 512 tokens.

<bos>: encode with it

The Gemma 4 E2B-it tokenizer adds no <bos>, but this model was trained with one at the start of every record. d1a from jonpol01/d1a#261 on (merged as b4b0f989) puts it back (bos_token_id 2; d1a_config.json records leading_ids: [2]). An app that encodes requests itself must prepend id 2 before right-padding to 512.

Run it

Needs d1a with jonpol01/d1a#261:

pip install coremltools numpy torch transformers safetensors "d1a @ git+https://github.com/jonpol01/d1a"
hf download JohnP1/d1a-e2b-it-coreml-4bit --revision v0.6 --local-dir d1a-e2b-it-coreml-4bit
python d1a-e2b-it-coreml-4bit/reference/run.py d1a-e2b-it-coreml-4bit request.json

request.json is a System One request, for example:

{"state": "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.",
 "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"returns": null, "shipping": null, "billing": null}},
               "urgent": {"type": "noul", "instructions": "Does this need a reply today?"}}}

The first run compiles the chunks into coreml/.compiled/ (a few minutes); later runs load them.

Files

Path	What
coreml/L512_<a>-<b>.mlpackage	5 chunks of the 35-layer text model, int4 weights (per block of 32). Chunk 7-15 outputs the shared keys and values that chunks 15-35 read.
embeddings.safetensors	the token and per-layer embedding tables (MLX affine-quantized, from JohnP1/d1a-e2b-it-mlx-q8@v0.6), read row by row on the host
head.safetensors, d1a_config.json	the pointer head, the temperature (1.41), the <bos>, pad and delimiter ids
config.json, tokenizer.json, tokenizer_config.json	the Gemma 4 E2B-it model config and tokenizer
io.json	each chunk's input names
reference/	Python reference: run.py (one request), workers.py (one process per chunk), d1a_coreml.py (host-side embeddings and masks), pipeline.py (all chunks in one process)

How a request runs

1. Encode it with D1A's layout (d1a.core.encoding.encode, d1a with #261, so it starts with <bos>, id 2) and right-pad it to 512 tokens with the pad token.
2. Look up the token embeddings (scaled by √hidden) and the per-layer embeddings (scaled by √256) on the host.
3. Build the block-causal masks (full and sliding), clamped to −1e4.
4. Run the chunks in order, each in its own process. Several large Neural Engine programs in one process can crash the runtime.
5. Read the hidden states at each question's <decide> and </opt> positions, score them with the head, divide by the temperature, and take the softmax.

macOS 15 / iOS 18 or later (block-wise int4 weights).

D1A-E2B-it builds

The same model in four builds. Mac checks on an Apple M1 Max, the same 28 requests (42 questions, all ≤ 512 tokens), each against the fp32 PyTorch rebuild of the 8-bit export:

Build	Repository	Runs on	Size	Mac check
MLX 8-bit	JohnP1/d1a-e2b-it-mlx-q8	Mac GPU	3.5 GB + 0.9 GB media	on its card
MLX 4-bit	JohnP1/d1a-e2b-it-mlx-q4	Mac GPU	2.5 GB + 0.9 GB media	on its card
Core ML int8	JohnP1/d1a-e2b-it-coreml	Neural Engine (CPU on M1)	3.5 GB (1.8 GB chunks + 1.6 GB embeddings)	on its card
Core ML int4	JohnP1/d1a-e2b-it-coreml-4bit	Neural Engine (M1: not checked)	2.6 GB (1.0 GB chunks + 1.6 GB embeddings)	p50 16.0 s, p95 55.6 s per request (28 requests), 338 s to load (first, compiled); 4 of 42 answers change (2 of 40 clear-cut ones, where the reference's top answer leads by more than 0.05); largest probability change 0.326.

License

Apache-2.0. Base model: Gemma 4 by Google (Apache-2.0). D1A is built on Kev by Jared Palmer (Apache-2.0).